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ABSTRACT 



This report is one of a series presenting findings from 
"Prospects: The Congressionally Mandated Study of Educational orowth and 
Opportunity." This study, conducted in response to the 1983 Hawkins-Staf f ord 
Amendments, was a major effort to examine the effects of Chapter 1 on student 
achievement and other school-related educational outcomes. Data were 
collected during school years 1991-1994 from nationally representative 
samples of students from grades 1, 3, and 7 in an initial sample of about 400 

schools. An essential finding of an interim report was that Chapter 1, as it 
was configured in 1991 and 1992, was insufficient to close the gap in 
academic achievement between Chapter 1 students and their more advantaged 
peers. In the period covered by this study, the achievement gap remained a 
reality. There were some highly disadvantaged schools in which children 
performed better than students in other high-poverty schools, and these were 
characterized by school -wide Chapter 1 programs; greater use of tracking by 
ability; more experienced principals; lower rates of student and teacher 
mobility; a balanced emphasis on remediation and higher-order thinking in 
classroom instruction; and higher levels of community, parent, and teacher 
support for the school's mission. Chapter 1 did serve the students most in 
need of help, but its assistance was insufficient to close the achievement 
gap. This is not to say that Chapter 1 was not helpful, but it was not enough 
to bring its students up to par. Data from the Prospects study support 
earlier research findings that the characteristics of the individual student 
and family account for the largest part of the variation in student 
achievement as measured by test scores, but that schools do make an important 
contribution that can be enhanced. Three appendixes present characteristics 



of low and high poverty schools, characteristics of high-performing 
high-poverty schools, and a description of the Technical and Stakeholder Work 
Group for the study. (Contains 13 exhibits and 9 references.) (SLD) 
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Executive Summary 

The Chapter 1 Program 

For more than 30 years, the federal government has provided financial assistance to 
schools to compensate for the effects of poverty on the educational progress of disadvantaged 
children. Chapter 1 (or Title I), the flagship program targeted at this objective, with funding 
of about $7 billion, reaches over six million children each year, covering about three-quarters 
of all elementary schools and nearly half of all middle and secondary schools, About 21-23 
percent of students in the 1st through 3rd grades participate in Chapter 1, declining to about 
18-21 percent for children in the 4th through 6th grades, and to 5-8 percent for students in the 
7th through 9th grades. During the early 1990s most children received targeted assistance with 
only a relatively small percentage of students attending a schoolwide program (a program 
option allowing all students in an eligible school to receive assistance). 

Despite its ambitious mandate, the program has been a relatively marginal educational 
intervention Until the recent, legislative changes passed in 1994, funds were spread thinly among 
a large number of eligible schools. In addition, services tended to be targeted toward the early 
primary grades, and most participating children received assistance in reading/language arts alone; 
relatively few students received additional instruction in math. Typically, Chapter 1 operated as 
an “add on” to regular classroom instruction and added only an average of 10 minutes of extra 
instructional time per day. Typically, Chapter 1 students were pulled out of their regular 
classroom for supplementary instruction during the school day. This method of service delivery 
has been criticized for stigmatizing students, disrupting regular classroom instruction, and 
reducing the time students spend with their regular classroom teacher and classmates. 

The level of instructional assistance Chapter 1 students generally received was in stark 
contrast to their level of educational need. On average, these children were poor, non-white, had 
undereducated parents, had parents for whom English was not their native language, and often 
lived in urban areas with the associated increased likelihood of exposure to violence and an array 
of modem social ills. Also, as expected, these children had substantial educational deficits — they 
scored well below their more advantaged peers on standardized tests, received low grades, and 
were rated lower by their teachers on a variety of important dimensions including attention, 
cooperativeness, effort and motivation, and participation in class. 



The Prospects Study 

This report is one of a series to present findings from Prospects : The Congressionally 
Mandated Study of Educational Growth and Opportunity. This study, conducted in response to 
the 1988 Hawkins-Stafford Amendments, was a major effort to examine the effects of Chapter 1 
on student achievement and other school-related educational outcomes. Data for the study were 
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collected during school years 1991-1994 from nationally representative samples of students 
selected from three cohorts of students enrolled in an initial sample of about 400 schools. Data 
collection began for lst-grade students in the fall of 1991, and for 3rd-grade and 7th-grade 
students in the spring of 1991 . Information about these students was collected annually using a 
variety of sources: standardized tests in reading and math; student surveys for students in the 3rd 
grade or above; abstraction of data in school records; parent surveys; and questionnaires 
administered to teachers, principals, and district administrators. Students in the 1st- and 3rd- 
grade cohorts were followed until spring, 1994; students in the 7th-grade cohort were followed 
until spring 1993. 

The Interim Report 

An interim report, Prospects: The Congressiomlly Mandated Study of Educational 
Growth and Opportunity, 1993 described participation in the baseline year 1991 and documented 
a number of important changes in student performance that occurred between 1991 and 1992: 

■ The learning gap between high- and low-poverty schools was large, and appeared 
to increase as children progressed through school. 

■ Chapter 1 participants were highly disadvantaged students on both academic and 
behavioral measures: their initial gap in performance did not change during their 
participation in compensatory education over the school year. 

■ Many children who were in need of extra assistance were not being served by the 
existing compensatory education programs. 

The essential finding that Chapter 1 , as it was configured at that time, was insufficient to 
close the gap in academic achievement between Chapter 1 students and their more advantaged 
peers contributed to a significant rethinking of the program. In 1994, this helped lead to 
important legislative changes to federal education support that included the following goals: 

■ Ensi: •>.• that children served by the program are expected to achieve the same high 
standards that the states apply to all their students. 

■ Support challenging curriculum and accelerated instruction to provide students 
with the opportunity to meet challenging state standards. 

■ Expand schoolwide programs to more high-poverty schools to improve the 
curriculum and instruction of entire schools. 

■ Help support high-quality teacher training to prepare teachers to provide 
challenging curriculum and instruction. 
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■ Promote partnerships between parents and schools. 

n Hold schools and districts accountable for showing progress in raising the 
performance of students served by the program. 

■ Target money to the neediest schools. 

Prospects data collection was ended at the time of the 1994 legislative changes to Chapter 
1 (which included renaming the program to its original name of Title I). This report presents 
results covering the entire study period from 1991-1994. 

Conclusions 

School Poverty Concentration Is Associated With Lower Academic Performance 

In the period covered by this study, children in high-poverty schools began school 
academically behind their peers in low-poverty schools, and were unable to close this gap in 
achievement as they progressed through school. 

When assessed against high academic standards, most students failed to exhibit the skill 
mastery in reading and mathematics expected for their respective grade levels. Students in high- 
poverty schools were, by far, the least able to demonstrate the expected levels of academic 
proficiency. 

Academic Standards Vary Between High- and Low-Poverty Schools 

A cornerstone of current educational reform initiatives is the desire to hold all students to 
high academic standards. Tliis objective may, however, pose serious difficulties for students in 
our most disadvantaged schools as higher standards could increase an already substantial gap 
between the achievement levels of the children of the poor and the well-to-do. For example, 
when measured against standardized test scores, a grade of “A” in a high-poverty school is 
equivalent to a grade of about “C” in a low-poverty school. 

There Are Some High-Poverty Schools That Show Better Student Performance 

There were at least a small number of highly disadvantaged schools in which students 
performed better than students in other high-poverty schools. These high-performing high- 
poverty schools were characterized by: a schoolwide Chapter 1 program, greater use of 
tracking by student ability, more experienced principals, lower rates of teacher and student 
mobility, a balanced emphasis on remedial and higher-order thinking in classroom instruction, 
and higher levels of community, parent, and teacher support for the school’s mission. 
Although we cannot make reliable attributions of the characteristics of these schools to thei. 
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ability to produce seemingly higher levels of student performance, the data are suggestive and 
provide a direction for future research. 

Chapter 1 Services Did Not Close The Gap In Student Academic Achievement 

Chapter 1 did, on average, serve those students who were clearly most in need of 
supplementary assistance. However, Chapter 1 assistance was, on average, insufficient to 
close the gap in academic achievement between advantaged and disadvantaged students. The 
observed lockstep pattern of student growth clearly demonstrated that where students started 
out relative to their classmates is where they ended up in later grades. 

Our inability to discern a compensatory effect of Chapter 1 is not necessarily an 
indication of program failure. Limitations of the Prospects study prevented us from observing 
directly whether Chapter 1 students would have been worse off (i.e., whether the gap would 
have widened over time) in the absence of the services they received. In fact, we might expect 
the gaps to grow over time, absent a special intervention. Chapter 1 may have helped but was 
too weak an intervention to bring the participating students up to par with their classmates. 

Factors Associated With Higher Student Performance In School 

Data from Prospects confirm the earlier findings by Coleman et al. (1966) that the 
characteristics of an individual student and his/her family account for the largest part of the overall 
variation in student achievement as measured by test scores. However, although relatively 
smaller, school factors do make an important contribution to student academic achievement and 
growth. Furthermore, all schools have students that exhibit a range of academic achievement 
levels; poor schools are not filled solely with low-achieving students, nor are low-poverty schools 
filled solely with high-achieving students. There are good students in most schools, even in the 
most troubled places, and even the best schools have students for whom academic work is a 
challenge. 

The statistical models used to assess the effect of Chapter 1 participation also provide 
evidence of factors that are important correlates of student achievement: 

■ Student and Family Characteristics: preschool education was associated 
with higher achievement; minorities and non-native English speakers 
exhibited lower achievement; girls had an early grade advantage in reading 
skills, but generally did less well in math throughout the grade span 
covered by this study; and students from poor families, and those who 
experienced frequent moves, performed less well academically. 

■ Parents: Parental expectations for their children’s’ success in school and 
parents’ involvement in school activities, were associated with higher 
student achievement. Surprisingly, we found that parents who were more 
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actively involved with their children’s school work were associated with 
lower student achievement. But this is likely related to the greater need for 
attention and supervision of low-achieving students 

■ Teachers and Schools. Higher levels of school poverty concentration were 

associated with lower student academic achievement, as were grade 
retention and higher levels of student disciplinary actions at the school (i.e., 

increased disruption). Teachers who placed a greater emphasis on 

comprehension and higher-order thinking skills were associated with higher 
student academic achievement. 
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CHAPTER ONE 

INTRODUCTION 



F or more than 30 years, since the passage of Title I of the Elementary and Secondary 

Education Act (SEA) in 1965, the federal government has provided financial 
assistance to improve education for elementary and secondary school children who are identified 
to be at risk of school failure, and who live in low-income communities. The original goals of 
Title I (later called Chapter 1 before regaining its original title in 1994), 1 were deeply embedded in 
the 1960's social reforms of the Great Society; 

■ To provide financial assistance to school districts where the concentration of low- 
income families reduced their ability to provide adequate educational programs; 
and, 

■ To meet the special educational needs of disadvantaged children. 

Since its inception, the program has been the cornerstone of federal support to education, 
and its annual appropriation of about $7 billion dominates the federal elementary and secondary 
education budget. The breadth of the program’s influence in public education should not be 
underestimated. It reached over six million children annually, primarily in the early elementary 
grades (one in every five 1st -graders participated in Chapter 1), three-quarters of all elementary 
schools, about half of middle and junior high schools, and one-quarter of high schools (Millsap, 
etai., 1992) 



1 Because this study was conducted during the period when the program was referred to as Chapter 1 , we will use this notation 
throughout this report. 
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The Prospects Study 2 



Congressional Mandate 

Congress, in the 1988 Hawkins-Stafford Amendments to the Elementary and Secondary' 
Education Act (Section 1462, Public Law 100-297), mandated a “national longitudinal” 
assessment of the impact of Chapter 1 based on a comparison of the “...educational achievement 
of those children with significant participation in Chapter 1 and comparable children v/ho did not 
receive Chapter 1 services.” The 1988 legislation further required that the study “...should be 
conducted throughout the country in urban, suburban and rural areas and shall be of sufficient size 
and scope to assess and evaluate the effect of the program in all regions of the country .” 

Study Design 

To meet this mandate, Prospects collected a large amount of information from students 
(standardized tests and self-administered surveys), parents, teachers (both regular classroom and 
Chapter 1 instructors), principals, district-level staff, and students’ school records. The sample 
was large (initially involving nearly 40,000 students in 365 schools), 3 it covered a broad grade 
span from 1st to 9th grade, and tracked the same students from 1991 to 1994. As shown in 
Exhibit 1.1, data were collected from three student grade cohorts: 

■ lst-grade cohort: students beginning 1st grade in the fall of 1991, who were 

tracked from entry into school through completion of the 3rd grade in the spring of 
1994; 



2 For a complete description of the Prospects study design and implementation sec Puma, Michael et ai, (1997), Prospects: The 
Congress ion ally Mandated Study of Educational Growth and Opportunity — Technical Report. Abt Associates Inc.: Bcthesda, 
MD. 

3 Samples of students who moved from their initial school during this field period were followed and interviewed and 
tested in their new schools By the end of the data collection period the total number of schools increased to 1 ,760 
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3rd -grade cohort: students enrolled in the 3rd grade in the 1990-91 school year, 
who were tracked from the end of their 3rd grade year (spring 1991) through 
completion of the 6th grade in the spring of 1994, and 

7th-grade cohort: students in the 7th grade during the 1990-91 school year, who 
were tracked from the end of 7th grade (spring 1991) through the completion of 
the 9th grade in the spring of 1993. 



No other currently available information allows analysis of the rich relationships among 
student characteristics and childhood experiences, family environment, and school and classroom 
characteristics and instructional processes. Data collected included the following: 



■ Student data were collected from several sources. The student abstract collected 
information from school records: absenteeism, tardiness, and grade retention; 
educational experiences such as preschool and Head Start attendance; identified 
disabilities; and participation in Chapter 1 or other remedial or special education 
programs. The student profile, completed by ti » teacher who was most 
knowledgeable about the student (usually the regjid.* classroom teacher), included 
teacher ratings of the student on topics such as ability, motivation, attitudes, 
classroom socialization, language skills, and health status. Finally, a student 
questionnaire, administered to students in the 3rd grade and above, collected data 
on current and past educational experiences, student and family demographics, 
educational aspirations, perceived academic strengths, grades, school attendance, 
participation in extracurricular activities within school and outside of school, and 
parent participation in their schooling. 

■ The parent questionnaire provided information on parent and family 
characteristics, and additional information about parental attitudes and practices 
related to student learning. Topics included parental demographic information, 
household composition, income and employment, and home environment 
characteristics such as family rules, educational activities in the home, aru parents' 
expectations for their child. 

■ Academic achievement was assessed using the Comprehensive Test of Basic 
Skills 4th edition, a vertically equated test series designed to measure achievement 
in the basic skills taught in schools throughout the nation The CTBS/4 was used 
to measure student achievement status and gains in reading/language arts and 
mathematics. 
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1st Grade Cohort 



Spring '91 Fall *91 Spring ‘92 Spring '93 

Survey - Assessment Dates 



i 



, 

Spring ‘94 



■ The teacher instruments included information about teacher background, 

certification and experience, school climate (e.g., parent contacts, interactions with 
administration, and degree of influence over policies), and classroom instructional 
practices. Regular classroom teachers and Chapter 1 teachers were asked about 
the characteristics of their respective programs (teaching load, grade assignments, 
frequency of instruction, use of aides, instructional techniques), their 
responsibilities (subjects taught and extracurricular activities), their access to 
resources (including use of computer-assisted teaching programs), and their 
assessment of the performance level of their students. Teachers were also 
questioned about the coordination of the Chapter 1 program with regular teaching, 
and the selection process for admitting students to Chapter 1 instruction. 
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The principal questionnaire and the characteristics of schools and programs 
(CSP) were both administered to the school principal or his/her designate. The 
principal questionnaire covered the following topics: the principal's background 
and experience, school policies, administrative techniques, and Chapter 1 program 
features (e g., goals of the program, selection procedures, and parental 
involvement). The CSP questionnaire focused on the organization of the school 
(enrollment, grades taught, size of staff, class organization), demographics of the 
student body, and type and organization of Chapter 1 programs (and other special 
education programs) at each grade level. 

Finally, district information was obtained from the Chapter 1 district coordinator. 
Topics covered on this questionnaire included: district staff size, enrollment at 
each grade level, daily attendance, length of the school day, provision of in-service 
trai ning for personnel, and demographics for the district (racial composition and 
poverty level) . The district coordinator was asked about the characteristics of the 
Chapter 1 program, including the selection process for schools and students, 
instructional approaches in each subject area, staff size, method of evaluating the 
program, and activities for parental involvement. The district coordinator was also 
asked about characteristics of non-Chapter 1 compensatory education programs 
and services for language minority and Limited English Proficient (LEP) students. 



Chapter 1 As It Operated in the Early 1990s 

Throughout its history, Chapter 1 has been essentially a funding source. Beyond some 
broad guidelines, local school districts and schools had enormous flexibility to decide where and 
how to focus the resources they received. That is, they could decide which schools and grades 
received funds, how much they received of the types of services that were provided, the content 
areas that were targeted, how teaching was done, and by whom. Consequently, the program 
could mean very different things in different schools. Furthermore, even for those students who 
received assistance, federal funding was equal to only about 1 5 percent of total average 
expenditures per pupil (Riddle, 1996). This small proportion of federal assistance is due to a 
variety of factors, including, 1) the limited level of funding in relation to the size of the eligible 
population, 2) allocation formulas that spread the funds thinly across a large number of school 
districts, and 3) an inadequate targeting of funds to the high-poverty schools where the level of 
need was greatest. 
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School and Student Selection 



The U.S. Department of Education distributed Chapter 1 funds directly to counties based 
on the number of poor children aged 5-17 identified in the decennial census, with an adjustment 
using a state average per-pupil expenditure factor intended to recognize cost of living differences 
across states States then suballocated the funds to school districts in proportion to the number 
of children from low-income families in each district. Within the school district, a school was 
eligible for Chapter 1 assistance if its attendance area had a poverty rate that was relatively high 
for that district. In general, districts selected the schools that ranked highest in poverty, but the 
law allowed districts several alternatives to a strict ranking. Finally, within selected schools, 
individual students were selected for services on the basis of their educational need, rather than on 
the basis of their parents’ financial status. According to Prospects, almost all students attended 
schools that used standardized testing to select students for Chapter 1 services. 

Staffing and Spending Patterns 

Districts reported that between 70 and 80 percent of Chapter 1 dollars were used to pay 
salaries for instructional, administrative, and support staff. However, staffing arrangements for 
Chapter 1 instruction differed for low- and high-poverty schools. 4 Although classrooms in low- 
and high-poverty schools had roughly the same number of staff per child, the high-poverty schools 
were more likely to use the funding to staff the classroom with additional instructional aides, in 
contrast, low-poverty schools tended to use certified teachers. The aides also assumed different 
responsibilities in low- and high-poverty schools. Aides in low-poverty schools were more likely 
to have responsibility only for non-instructional tasks while aides in high-poverty schools had 
responsibilities for instructional as well as non-instructional tasks. As a consequence, the extra 
resources provided to high-poverty schools by Chapter 1 may not have necessarily improved the 
quality of classroom instruction on a par with the more advantaged schools. 



4 Throughout this report, school poverty is measured using the percentage of enrolled students who were eligible for free and rcduced-pncc school 
meal*. High -poverty schools six those in which 75 percent or more of the students are eligible; low-poverty schools are those in which 25 
percent of fewer of the students are eligible. 
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Arrangement of Chapter 1 Services 



The Prospects data match earlier research findings that show a declining reliance on pull- 
out as the only setting for Chapter 1 services (i.e., where students are removed from their regular 
classroom during the school day for supplemental instruction). But this method of instruction still 
remained the dominant mode of service delivery, especially in the low-to moderate-poverty 
schools. For example, about three-quarters of all 3rd-grade Chapter 1 students received seme es 
by being pulled out of their regular classrooms. This mode of service delivery has been often 
criticized for disrupting the regular classroom, for stigmatizing students who are pulled out, and 
for providing instruction of uneven quality. 

Subject Matter and Services 

Most students who participated in Chapter 1 received assistance in reading/language arts. 
Only some students received math instruction, and relatively few Chapter 1 participants received 
services in non-instructional areas such as counseling or health education (about 3 percent). For 
example, of the students participating in Chapter 1, 96 percent of lst-graders, 83 percent of 
4th-graders, and 8 1 percent of 8th-graders received assistance in reading. In contrast, about 30 
percent of lst-graders received services in both reading and math; the corresponding figures for 
the 4th and 8th grades are 37 and 22 percent, respectively. However, a greater percentage of . 
students in high-poverty schools received assistance in both reading and math than did 
participants in low-poverty schools. 

In elementary grades, Chapter 1 math teachers and regular classroom teachers typically 
used the same instructional materials at the same grade level. However, Chapter 1 reading 
teachers more often reported the use of different materials at the same instructional level. It was 
also found that the availability and use of computers by teachers in classrooms varied greatly by 
grade, with middle school students having the lowest access to teachers who used computers. 
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Finally, the dominant practice in Chapter 1 math was whole class instruction; in Chapter 1 
reading, within-class grouping was used slightly more often than whole class instruction. 



Instructional Time 



The majority of Chapter 1 students received supplemental services five days per week for 
an average of 30 minutes per day. With regard to the rest of students’ time in school, Prospects 
data indicate the following: 



■ High-poverty schools allocated more time to Chapter 1 instruction than low- 
poverty schools. 

■ Overall, substantially more time was allocated to reading than to math instruction 
for students in the 1 st and 4th grades; by the 8th grade about the same amount of 
time was allocated to both subject areas. 

■ High-poverty schools allocated more time for reading and math instruction than 
low-poverty schools in the 1st and 4th grades. However, by the 8th grade, 
students in high-poverty schools received less instructional time in reading and 
math than students in low-poverty schools. 

■ Students in high-poverty schools were more likely to have opportunities that 
increase learning time outside of school (e.g., before/after school programs and 
summer school) than students in low-poverty schools. 

■ Students taught using in-class arrangements received about five minutes more per 
day of academic instruction than students in pull-out programs. 



Curriculum and Instruction in Regular Classrooms 

Chapter 1 compensatory services comprise only a fraction of a student’s school day. The 
purpose of this section, then, is to better understand what happens in the regular classroom, and 
to compare the curriculum and instruction in low- and high-poverty schools: 
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■ High-poverty schools were more likely than low-poverty schools to rely on a 
traditional approach to reading instruction. This approach emphasized reading 
readiness and decoding, the use of multiple instructional groups, and the use of 
textbooks and basal readers. 

■ Whole class instruction was the dominant practice in all schools. First-grade 
reading instruction was the only situation in which appreciable grouping was used 

■ When grouping was used, the basis for grouping was most often similar student 
abilities. 

■ Students' regular math and reading classes contained an average of about 22 
students, with little variation by school poverty. Rarely reported were class sizes 
approaching levels found to be associated with differences in student learning (see 
Mosteller, Light and Sachs, 1 966) 

■ Tutoring was most often carried out as peer tutoring, followed by tutoring using a 
certified teacher. The use of paraprofessionals to tutor was most frequently found 
in high-poverty schools. 

■ For both reading and math instruction, teachers of students in high-poverty schools 
were most likely to report that computers were never used in their regular 
classrooms. However, most students, even in high-poverty schools, were in 
classrooms where computers were used at least some of the time. 



Coordination of Services 

Most educators believe coordination of services between Chapter 1 and the regular 
classroom instruction is critically important if students are to gain the most from their school day. 
The most frequently reported means of communication for this purpose was formal discussion 
between the two teachers. Further, Chapter 1 and regular classroom teachers agreed that the 
primary responsibility for the student's instruction and progress rested with the regular classroom 
teacher. 
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Characteristics of Chapter 1 Participants 



It is important to understand the characteristics of the participating students to appreciate 
the challenge inherent in the Chapter 1 goal of “closing the gap” in academic achievement 
between them and their more advantaged classmates. While a more complete picture is available 
elsewhere, 5 the key findings from Prospects are the following: 



■ Compared to their nonparticipating classmates, Chapter 1 participants were more 
economically disadvantaged. They were more likely to be living with a single 
parent, have a total family income under $10,000, receive public assistance, and 
have poorly educated parents whose nature language is not English. They were 
also more likely to be non-white and live in urban areas with the concomitant 
increased likelihood of being exposed to high rates of crime, physical violence, 
drug abuse, and substandard living conditions. 

■ Not surprisingly. Chapter 1 participants were also educationally disadvantaged. 
They scored poorly on reading and math achievement tests, received low grades, 
were more likely to be retained in grade, and received low ratings of their ability 
from their classroom teachers. Chapter 1 students also did less reading outside of 
school, and were less inclined to use the public library. 

■ The teachers of Chapter 1 children were also more likely to identify a variety of 
problems that made these students more at risk for school failure compared to their 
classmates. Chapter 1 students were reported to exhibit lower effort in school, be 
less motivated, be less likely to follow rules and directions, exhibit less creativity 
and independence, be less mature, and be less likely to complete school work and 
to participate in class. The Chapter 1 students themselves were also more likely to 
report that they had negative experiences at school, including disciplinary action 
and having their parents called to school. 

■ Rates of Chapter 1 participation (targeted and/or schoolwide assistance) were 
generally constant over the grade span of each of the three study cohorts (see 
discussion below) but declined across cohorts. About 21-23 percent of lst-grade 
cohort students, 18-21 percent of 3rd-grade cohort students, and only 5-8 percent 
of 7th-grade cohort students participated in Chapter 1 . In the first half of the 



5 Puma, Michael ct ai. ( 1 993) Prospects: The Congressionally Mandated Study of Educational Growth and Opportunity , Abt 
Associates Inc.: Bethesda, MD 
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1990s, most students received targeted assistance — schoolwide programs covered 
only 3-5 percent of 1st- and 3rd-grade cohort students, and well under 1 percent of 
7th-grade cohort students. 

■ Among nonparticipating students, only 3-10 percent participated in another form 
of compensatory education. For many students, particularly those in upper grades, 
services were simply not available as schools had elected to focus their attention 
on earlier grades. Services were found to be unavailable for 26-32 percent of 1 st- 
and 3rd-grade cohort students, and 47-58 percent of 7th-grade cohort students. 

■ Although our data do not cover all years of individual students’ schooling, it does 
appear that most Chapter 1 students received assistance for relatively short periods 
of time. About half of 1st- and 3rd-grade cohort participants received assistance 
for 1-2 years, and nearly all 7th-grade cohort participants had periods of 
participation lasting from 1-2 years. For the most part, program availability was 
not found to be the reason for students leaving the program or having interrupted 
periods of Chapter 1 assistance. 

■ As previously noted, most Chapter 1 students received help in reading/language 
arts/English alone or in combination with assistance in math. Relatively few 
received only math instruction. 



Past Accomplishments and Shortcomings of Chapter 1 

During the two decades prior to 1990, test scores, high school graduation rates, and 
college attendance rates improved dramatically for students from low-income and minority 
famili es From 1 970 to the late- 1 980s, the learning gap between whites and minorities was cut 
by over one-third in reading, mathematics, and science as measured by test scores on the National 
Assessment of Educational Progress (NAEP). While these gains cannot be directly attributed to 
federal programs alone, a recent RAND report asserts that these initiatives contributed to the 
educational improvement of disadvantaged and minority students. According to the authors, 
policies aimed at equal educational opportunity for disadvantaged students and increased public 
investments in their schools and families “provide the most likely explanation for the gains made 
by black and Hispanic stidents over and above those predicted by family effects” (Grissmer et al., 
1994, p. 107). 
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However, more recent data from NAEP, the Prospects Interim Report (Puma, et al., 

1993), and other evaluations conducted for the National Assessment of the Chapter 1 Program 
(U.S. Department of Education, 1993) showed that progress in closing the achievement gap had 
stalled. The National Assessment concluded that the Chapter 1 program, as then structured , was 
insufficient to help students continue to improve and make up the early gaps in academic 
achievement. Specifically, the report concluded that: 



■ The progress of Chapter l participants on standardized tests was no better 
than that of nonparticipants with similar backgrounds and prior achievement 
levels. 

■ The poverty level of certain schools placed disadvantaged students in double 
jeopardy. School poverty depresses the scores of all students in schools where at 
least half of the students are eligible for subsidized lunch, and seriously depresses 
the scores when over 75 percent of students live in low-income households. 

■ Many Chapter 1 programs reinforced low expectations for student 
achievement Students in high-poverty schools were exposed to a “watered- 
down” and non-challenging curriculum when compared to other students. Data 
from Prospects , as shown in Exhibit 1 .2, indicated that academic standards in high- 
poverty schools fell far short; performance that rated an “A” in nigh-poverty 
schools, was rated about a U C” in low-poverty schools when assessed against 
standardized test scores. 

■ Chapter 1 often operated as an add-on program that worked on the margins, 
adding only an average of 10 minutes of extra instructional time per day. 

* When carried out as a separate add-on program, Chapter 1 did not work to 
improve the regular program of instruction, where children in Chapter 1 spent 
almost their whole day. 

■ A focus on compliance and regulatory matters, rather than an emphasis on 
improved student performance, occupied much of states’ and districts’ efforts in 
administering Chapter 1 

■ Chapter 1 dollars were spread to 71 percent of public elementary schools, 

including half of all elementary schools serving fewer than 1 0 percent poor 
children. Meanwhile, many high-poverty high schools and middle schools 
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frequently went unserved as districts focused their funds on elementary schools- 
including those with lower poverty rates. 

These results, and corroborating evidence from other research studies, were instrumental 
in shaping several changes to the program in 1994. These changes included provisions that 
would: 

m Ensure that children served by the program are expected to achieve the same 
high standards that the states apply to all their students. Student assessments 
that are aligned with academic standards will measure how well Title 1 students are 
progressing in meeting challenging standards. 

■ Support challenging curriculum and accelerated instruction. Poor and 
minority students should have access to the same challenging curriculum as other 
students. Title I resources are expected to be used to provide participating 
students with the extra instructional time and services they need to meet new 
expectations for high achievement. 

* Expand schoolwide programs to more high-poverty schools to improve the 
curriculum and instruction of entire schools. The schoolwide option gives 
schools the flexibility to use Title I and other funds in ways that address the needs 
of all students. 

■ Help support high-quality teaching training. Expanded efforts are to be 
implemented to increase the capacity of teachers to provide challenging curriculum 
and instruction. 

■ Promote partnerships between parents and schools. Additional support to meet 
the needs of children will come through parent-school compacts, and increased 
support for family involvement and family literacy programs. 

■ Hold schools and districts accountable for showing progress in raising the 
performance of students served by the program. After assisting in school 
improvement, school districts and states are to impose sanctions on schools that 
have been chronically identified as failing 

■ Target money to the neediest schools, including middle and high schools. 
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This Report 



This report is intended to serve as a non-tecbnical description of the Prospects findings. 
Chapter Two describes the differences in student academic achievement between high-and low- 
poverty schools, and examines the effect of the Chapter 1 program on student educational 
growth Chapter Three then compares the characteristics of high- and low-poverty schools, and 
describes the characteristics of high-performing high-poverty schools, i.e., those where students 
exhibit better academic achievement when compared to other high-poverty schools. Finally, 
Chapter Four presents an exploratory examination of the relationship between student, family, and 
school variables and student educational outcomes. 
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CHAPTER TWO 



DID CHAPTER 1 HELP CLOSE THE GAP? 

T he primary objective of Chapter 1 is to help disadvantaged students overcome the 
negative effects of poverty and, in particular, to help close the educational 
achievement gap between them and their more advantaged classmates. This chapter seeks to 
answer the question of the extent to which the program was able to achieve this important 
objective. 

Throughout this discussion it is important to keep in mind that the reported results 
represent the Chapter 1 program as it existed nationally during the period from 1991 to 1994. 
These are nationally representative findings that describe the typical experiences of Chapter 1 
students. These results do not reflect the way the program could operate, or indeed, may operate 
today as Title I , just the way it was implemented during the period of our study. 

Methodological Issues 

We begin our examination with a simple descriptive analysis of the academic outcomes 
used in the Prospects study. 1 That is, we examine the year-to-year changes in our selected 
student outcome measures not controlling for factors that can affect how these temporal patterns 
vary among students and/or between schools. As part of this exploration we describe how these 
growth rates vary among students in low- and high- poverty schools, and among students who do 
and do not receive Chapter 1 assistance. The second half of this chapter expands this descriptive 
analysis by using sophisticated statistical models that both control for a variety of important 



1 A complete description of the Prospects data, and the analysis methods used throughout this report, can be found in Puma, 
Michael, et al., (1997), Prospects: The Congressional ly Mandated Study Of Educational Growth And Opportunity — 
Technical Report , Abt Associates Inc.: Bethcsda, MD. 
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student, family, and school characteristics, and take advantage of the longitudinal and multi-level 
(i.e., students are clustered within classrooms and schools) nature of the Prospects data. The 
analyses reported here and in subsequent chapters are based on a longitudinal analysis sample 
constructed from the Prospects data That is, the same students were followed across all years of 
data collection. In addition, sophisticated imputation methods were used to deal with the 
inevitable problems of missing data 

For the purposes of these analyses, three types of student outcome measures were used 

■ Standardized scale scores in Reading Vocabulary, Reading Comprehension, 
Math Concepts and Applications, and Math Computation (all measured by 
the annually administered CTBS-4 test); 

■ Teacher ratings of student achievement in reading, language arts/English, 
and math; and, 

■ Teacher ratings of student behavior including classroom participation, 
cooperativeness, and attention (these are multi-item empirically derived 
scales). 

Across these different student outcomes measures, the concept of “growth” cannot be 
separated from the nature of the test or scale itself. For example, the teacher rating scales are 
affected both by the contextual setting (i.e., individual student ratings are conditioned by a 
teacher’s experience with other students in the same school) and changing expectations across 
grade levels. With regard to the first point, teachers in high- and low-poverty schools may use a 
different basis forjudging student achievement. For example, as noted in Chapter One, using the 
CTBS scores as a common indicator of achievement and class grades as a measure of teacher 
rating shows that students who received an “A” in a high-poverty school would receive a grade of 
about “C” in a low-poverty school. Additionally, the teacher ratings are not referenced to a 
common standard of expected performance, particularly the ratings of student behavior where the 
concept of “growth” is especially difficult. At each grade level, teachers are presumably rating 
students on the basis of their expected standards of behavior which are likely to change as 
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children mature. That is, the expected level of classroom participation, cooperation, and attention 
is obviously different for a 7th-grade child compared to a student in the lst-grade. As a 
consequence, we focus our discussion in this chapter primarily on student test scores. 2 

Similarly, the CTBS tests have an implicit growth trajectory that reflect an expected rate 
of learning from year to year. For our purposes we have based our analysis on scale scores that 
are, according to CTBS, “....units of a single equal-interval scale that is applied across all levels of 
the CTBS/4 regardless of grade or time of year of testing.” This equal-interval and “vertical 
equating” of the test makes scale scores suitable for statistical purposes, particularly for 
comparison among classes, schools, districts or other categories of students. For example, a gain 
of 50 scale score points has the same meaning regardless of where on the scale it occurs, or the 
difference in grade space over which the growth is observed (i.e., a 50 point gain from the 1st to 
the 3rd-grade has the same meaning as a 50 point gain from the 3rd to the 5th grade). 

Although scale scores can be statistically compared as noted above, some critics have 
raised concerns about comparing gains across grades for different groups of students because the 
test means for the national norm samples increase over grades, and the variability (i.e., the 
standard deviations) is not constant across grades. As a consequence, in the discussion that 
follows we have elected to describe any observed differences, both in absolute (i.e., the number of 
scale-score points separating two groups) and relative terms (i.e., differences expressed relative to 
the standard deviation of the CTBS norming sample). 

Relationship of Test Scores to School Poverty 



Exhibit 2. 1 displays plots of average CTBS scale scores, by reading and math sub-test, for 
four different groups: 



The accompanying Technical Report provides results on all the student outcome measures 
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Exhibit 2.1 Mean CTBS Scale Scores, by School Poverty Concentration and Grade 
(Weighted Means) 






Exhibit 2.1: (continued) 
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Exhibit 2.1: (continued) 
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■ The CTBS national test norms, 

■ The total weighted mean derived from the Prospects longitudinal analysis sample 
(labelled “all schools”); 

m Weighted means for students in high-poverty concentration schools, and 

■ Weighted means for students in low-poverty concentration schools 



Along the horizontal axis are the different test grades for each of our three study cohorts. 3 

Comparing average scores for high- and low-poverty schools shows, as expected, that 
children in the poorest schools achieved, on average, substantially below students in the more 
advantaged schools. For example, in reading vocabulary the achievement differences or "gaps” 
amounted to 52 scale-score points at the end of 1st grade, 46 points at the end of 3rd grade, and 
43 points at the end of 7th grade. These are very large differences equal to 0.84, 0.94, and 0.98 
standard deviations (SD) 4 respectively. The results for the other subtests tell a similar stoiy: 



■ Reading Comprehension— 56 points (0.72 SD), 61 points (1.05 SD), and 
34 points (0.77 SD) at the end of 1st, 3rd, and 7th grade respectively. 

■ Math Computation— 34 points (0.63 SD), and 40 points (0.83 SD) at the end of 
3rd and 7th grade respectively (this subtest was not administered at the start of 1st 
grade). 

■ Math Concepts and Applications-65 points (0.93 SD), 58 points (1.0 SD), and 
37 points (0.88 SD) at the end of 1st, 3rd, and 7th grade respectively. 



3 The labels on the horizontal axis represent the difTcrcnt measurement points for each of the three student cohorts. For example, 
1 st (S94) is the spring 1994 measurement point for the 1st grade cohort, or the end of the 3rd grade for these students. 

4 Measured in terms of the CTBS norm sample for the test used at each grade level. A difference of one standard deviation is 
generally considered to be a large effect (Cohen, 1 969) 
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Did these differences in average achievement diminish over time? As also shown in 
Exhibit 2.1, the answer is clearly no. With regard to each of the reading and math achievement 
tests we find the following growth patterns: 



■ Reading Vocabulary— the 
achievement gaps slightly 
increased in each cohort, 
from 52 to 54 points in the 
lst-grade cohort (0.84 to 
1 . 1 SD), from 46 to 54 
points in the 3rd-grade 
cohort (0.94 to 1.2 SD), and 
43 to 44 points in the 7th- 
grade cohort (0.98 to 1 .02 SD). 

■ Reading Comprehension-guys increased in the lst-grade cohort from 56 to 67 
points (0.72 to 1.16 SD), decreased in absolute terms in the 3rd-grade cohort 
from 61 to 48 points (but increased in relative terms from 1 05 to 1.12 SD), and 
remained constant at 34 points in the 7th-grade cohort (but also increased in 
relative terms from 0.77 to 0.87 SD). 

■ Math Computation-ihc achievement gaps remained unchanged in absolute terms 
in the 3rd-grade cohort at 34 points (while increasing in relative terms from 0.63 
to 0.74 SD), and increased from 40 to 50 points in the 7th-grade cohort (0.83 to 
0.98 SD). 

■ Math Concepts and Applications-g&ps remained essentially unchanged in 
absolute terms at 65 points in the 1st grade (increasing in relative terms from 0.93 
to 1.16 SD), decreased in absolute terms in the 3 rd-grade cohort from 58 to46 
points (but again, increasing in relative terms from 1 0 to 1 .07 SD), and increased 
in the 7th-grade cohort from 37 to 48 points (0.88 to 1.17 SD). 



Children in high-poverty schools began school 
behind their peers in low-poverty schools. This 
initial gap in academic achievement remained 
essentially unchanged as students in high- 
poverty schools moved into higher grades. 



These results clearly demonstrate that children in high-poverty schools, on average, 
started out academically behind students in low-poverty schools and were generally unable to 
decrease this gap in achievement as they progressed through school. 
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Further information on this gap in 
performance can be derived from the 
criterion-referenced test data provided by 
the CTBS. As depicted in Exhibit 2.2, even 
by the end of our survey period, the 
majority of children in all schools were 

unable to master the reading and math skills expected of them at their respective grade levels. 

For example, even in the least disadvantaged schools by the end of 9tii grade, well below one- 
third of the students were generally able to exhibit mastery of the skills reflected in the CTBS. 

These results are similar to those reported by the National Assessment of Educational Progress 
(U.S. Department of Education, 1994), 

More important for our purposes is the extent to which students in high-poverty schools 
were able to master these skills compared to students in low-poverty schools. As Exhibit 2.2 
shows, students in high-poverty schools were far less likely to exhibit skill mastery than students 
in low-poverty schools, and the performance gaps were generally larger in the higher-order skills 
than in the more basic skills areas. 

Student Outcomes and Chapter 1 Participation 

We now turn our attention to achievement levels and growth rates for students selected 
to receive Chapter 1 assistance and compare them to the achievement of nonparticipants. We 
begin with an examination of simple mean differences, and then extend the analysis using a 
variety of statistical models. 



When assessed against high academic 
standards , most students failed to exhibit 
mastery of skills in reading and mathematics. 
Students in high-povert y schools were, by far, 
the least able to demonstrate expected levels of 
proficiency. 
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Exhibit 2.2: Criterion Referenced Reading And Math Test Scores 5 by School Poverty and Grade Cohort: 
Percent of Students Mastering Each Skill Area (Weighted Means) 
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READING 














Word Meanings 


13.8 


10.4 


94 


4.8 


53.9 


15.2 


Multiple Word Meanings 


38.0 


10.4 


34.3 


4.8 


20.2 


3.5 


Affixes 


7.6 


2.6 


16.7 


4.2 


N/A 


N'A 


Words in Contcrxt 


76.5 


34.2 


25.4 


4.1 


28.5 


9.1 


Passage Analysis 


68.7 


29.4 


35.0 


7.3 


29.0 


5.0 


Central Thought 


55.4 


18.6 


294 


5.6 


27.9 


5.3 


Written Techniques 


55.7 


23.1 


237 


4.5 


19.7 


4.3 


Critical Assessment 


58.0 


21.8 


.34.5 


7.6 


18.1 




MATHEMATICS 














Addition 


78.0 


54.5 


49.8 


38.3 


N/A 




Subtraction 


46.5 


21.4 


65.5 


44.1 


WA 




Multiplication 


85.0 


66.9 


61.7 


37.5 


N/A 




Decimals 


68.2 


43.3 


54.7 


25.6 


42.7 


14.3 


Numcrawon 


69.1 


27.0 


40.9 


10.1 


21.3 


4.6 


Number Theory 


56.2 


21.4 


18.2 


4.2 


9.2 


1.9 


Data Interpretation 


67.8 


25.8 


37.4 


9.5 


20.8 


3.8 


Pre-Algebra 


63.4 


24.6 


32.5 


8.5 


20.9 


3.2 


Measurement 


62.7 


23.2 


39.2 


10.2 


16.3 


2.6 


Geometry 


30.1 


6.9 


29.3 


8.4 


9.8 


3.3 



5 It is important to note that the skills being assessed change over the different levels of the CTBS and are set by panels of outside 
experts relative to what children should know at a particular grade. Therefore, skill areas with the same name generally represent 
different levels of achievement or learning at different grades. 
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Descriptive Analysis 



Exhibit 2.3 contrasts average CTBS scale scores for children who never received 
Chapter 1 assistance with those who received supplemental help in at least one year during our 
study period. Not surprisingly, students who received Chapter 1 assistance were selected for this 
help because they scored low on standardized achievement tests, and our data clearly show this 
gap in academic ability. For example, if we focus on children in high-poverty schools (the main 
target of Chapter 1 assistance), the average gaps in Reading Vocabulary and Reading 
Comprehension ranged from 38 scale score points (0.61 SD) to 43 score points (0.55 SD) at the 
end of 1st grade, from 29 to 30 points (0.59 to 0.52 SD) at the end of 3rd grade, and from 24 to 
26 points (0.55 to 0.59 SD) at the end of 7th grade. 

Although not as large a gap compared to reading skills, the gap in math achievement is 
a’so large. The difference between Chapter 1 participants and nonparticipants in high-poverty 
schools was 32 points (0.46 SD) in Math Concepts and .Applications, and 17 points in Math 
Computation (0.25 SD). At the end of 3rd grade the differences were 28 points (0.5 SD) in Math 
Concepts and Applications, and 20 points (0.37 SD) in Math Computation; by the end of 7th 
grade the differences were 16 points (0.38 SD) and 6 points (0. 13SD) respectively. 

By the end of our study period 
(the end of 3rd grade for the lst-grade 
cohort, the end of 6th grade for the 3rd- 
grade cohort, and the end of 9th grade 
for the 7th-grade cohort), the 
achievement gaps were essentially the 
same in absolute terms, and slightly 
larger in relative terms. For example, in the lst-grade cohort, the test score differences in 
Reading Vocabulary remained at 38 points (increasing from 0.61 to 0.78 SD); and at 43 points in 
Reading Comprehension (increasing from 0.55 to 0.76 SD); 



Not surprisingly, students selected to receive 
Chapter l assistance scored far lower than their 
classmates on standardized tests of reading and 
mathematics. This gap in academic achievement was 
maintained as the children progressed through 
school. 
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Exhibit 2.3: Mean CTBS Scale Scores, by Chapter l Participation and Grade Level 
(Weighted Means) 





Exhibit 2.3: (continued) 
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dropped from 32 to 3 1 points in Math Concepts and Applications (increasing from 0.46 to 0.55 
SD); and rose from 17 to 19 points in Math Computation (increasing from 0.25 to 0.35 SD). The 
patterns in the other grades are largely the same as that observed for the 1 st-grade cohort. 

For the most part, then, students selected to receive Chapter 1 assistance were performing 
below their classmates and this achievement gap remained unchanged over time This finding is 
explored in more detail below where we attempt to statistically adjust for differences between 
Chapter 1 participants and nonparticipants. 

Multivariate Analysis 

The previous descriptive information on differences in academic gams between Chapter 1 
and non-Chapter 1 students is problematic because of the inherent noncomparability of these two 
groups of students. Chapter 1 students were significantly poorer, attended schools with higher 
concentrations of poor children, and, by the very nature of how they were selected for assistance, 
scored lower on standardized assessments of academic achievement than nonparticipants. As a 
consequence, a fundamental problem with an analysis of the impact of Chapter 1 is what 
researchers call “selection bias.” In this instance, students who received Chapter 1 services 
typically did so because they were having difficulty in school. Therefore, it is not surprising that 
we find that program participants did worse than nonparticipants. 

The difficulty is in finding an adequate comparison group for program participants that 
will allow us to determine what would have happened to Chapter 1 participants had they not 
received supplemental academic assistance Ideally, there would have been random assignment to 
control and treatment groups, but such a research design was not possible for this study. 

As an alternative, we have attempted to compensate for the absence of random assignment 
by using statistical models to “control for” measurable differences between Chapter 1 participants 
and nonparticipants. Our intent was to use these various characteristics to statistically adjust non- 
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Chapter 1 student outcomes so that they accurately represent what would have happened to 
participants absent the program. Such methods will not, however, produce the same kind of 
evidence regarding program impacts as would be possible from a study using random assignment. 
The statistical models represent our best attempt to establish a basis from which to judge the 
efficacy of Chapter 1 . (We will have more to say on this topic at the end of this chapter.) 

The statistical approach used, hierarchial linear modeling (HLM), was selected because it 
properly reflects the multilevel and longitudinal nature of the Prospects data to accurately model 
changes in student outcomes over time. This type of analysis is appropriate when, as in the case 
of Chapter 1, observed student outcomes are nested in a naturally occurring hierarchy of levels, 
For example, students were measured at different time points, students were nested within class- 
rooms, classrooms within schools, and schools within districts. 6 

In this particular application, we focused on three levels of this hierarchy. 7 At the first 
level, outcomes were modeled as a student's individual growth trajectory (i.e., using repeated 
measures of student outcomes over time). At the second level we captured differences among 
individual students in their background and family characteristics, as well as in the types of 
classroom instruction they received. The third level consisted of students nested within schools 
and captures variations across schools (district characteristics are also incorporated at this level). 
The statistical results at each level provided two important estimates: 



6 The main advantage of this approach is that it provides the ability to model processes within students (over time), between 
students, and between schools simultaneously, with maximum statistical precision at each level Wc therefore have a single 
analysis which models complex processes as they naturally occur. In addition, by estimating separate effects at each level of 
analysis, we avoid the problem of aggregation bias. 

7 It is conceptually possible to think of the Prospects data in terms of a four-level model with individual observations at the first 
level, the repeated observations across students at the second level, students clustered within classrooms at the third level, and 
classrooms clustered within schools at the fourth level. We did, in fact, initially begin our analysis with this structure in mind. 
However, because students changed classrooms every year, and because of the size and complexity of the data set, this type of 
four-level model was not computationally feasible. Our analysis has indicated that ignoring the classroom level of clustering is 
likely to have a small or ncgliblc effect on the estimated standard errors. 
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■ average level, which is defined as the mean (or average) score at the 
student’s first measurement point; and, 

■ slope, or the rate of change in level over time (i.e., across the different 
measurement points). 

For our purposes, we are more interested in the estimated slope because the primary objective of 
Chapter 1 was to increase the growth slopes for participants to allow them to catch up with 
their more advantaged classmates. Furthermore: 



■ Separate models were estimated for each of the three study cohorts. 

■ Within grade cohorts, separate analyses were conducted for each of ten 
different student outcome measures: CTBS scale scores for Reading 
Vocabulary; CTBS scale scores for Reading Comprehension; CTBS scale 
scores for Math Concepts and Applications; CTBS scale scores for Math 
Computation; teacher ratings of student achievement in reading; teacher 
ratings of student achievement in language arts/English; teacher ratings of 
student achievement in math; teacher ratings of student classroom 
participation; teacher ratings of student cooperativeness; and teacher 
ratings of student attentiveness. 

■ The time points entered into the models were as follows: lst-grade cohort: 
fall 1991 (in certain analyses only), spring 1992, spring 1993, spring 1994; 
3rd-grade cohort: spring 1991, spring 1992, spring 1993, spring 1994; and 
7th-grade cohort: spring 1991, spring 1992, spring 1993. 

■ Two measures of Chapter 1 program participation were used: 1) the 
number of years of Chapter 1 assistance a student received, and, 2) a series 
of variables that distinguished among various patterns of Chapter 1 
participation to determine whether students receiving services early in 
elementary school achieved more over time relative to students who 
received either no services or services in later grades. For the first measure 
of program participation, we focused our attention on the 1st and 3rd 
grade cohorts because of the relatively small number of Chapter 1 
participants in the 7th-grade cohort. For the second measure, we limited 
our attention to the lst-grade cohort to reduce the effect of early grade 
participation. 

■ Student outcomes were assessed in relation to the type of Chapter 1 
assistance a student received. For example, in the models of reading 
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f 

achievement, the Chapter 1 participation variable represented whether the 
student received assistance with reading skills. 

Four types of analyses were conducted using this statistical method: 



■ The first included a wide range of student, family, community, school, and 
classroom characteristics in addition to the previously described measures 
of Chapter 1 participation. 

■ The second approach to examining the effect of Chapter 1 added another 
covariate to the model, i.e., the students’ initial fall 1991 test score. This 
provided an additional level of statistical control to test Chapter 1 program 
effects independent of the students’ initial test score. But, because this test 
was only available for the lst-grade cohort our analysis was limited to 
these students. 

■ The third approach involved creating a “presumptively eligible” group of 
students entering 1 st grade defined as those students attending schools with 
50 percent or more of the children eligible for free or reduced-price meals, 
and who scored at or below the 25th percentile on the fall 1991 CTBS 
test We then compared the subsequent outcomes for students in this 
eligible group who did and did not receive Chapter 1 assistance. Chapter 1 
nonparticipants should be better matched to participants in this more 
narrowly defined comparison group. 

■ For the fourth approach, we focused on repeated measures of outcomes for 
individual students who received services in some, but not all, years of the 
study. This appr "*ach was intended to test the extent to which students’ 
standardized test scores were higher in years in which they received 
Chapter 1 assistance compared to years in which they did not. This 
analysis was done for both the 1st- and 3rd -grade cohorts. 



Initial Analysis of the Effect of Chapter 1 Participation 



We now move to our analysis of the effect of Chapter 1 participation on students. As 
noted above, our first statistical approach involved adding indicator variables measuring different 
amounts of Chapter 1 program assistance to hierarcliical linear models that controlled for a variety 
of student and school characteristics. If we were able to completely control for the differences 
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between participants and nonparticipants by including these variables in our model, the influence 
of these variables would show the true impact of Chapter 1 services. In this scenario we would, 
for example, hypothesize that if two students were equivalent on all measured variables (including 
their prior test scores), and if one of them received Chapter 1 assistance for an additional year, 
then the student who received more help should do better than the student who received less 
help. That is, if all other things were held equal by the model, getting more help should produce 
greater gains in the selected outcome measures than receiving less help. 

Exhibit 2.4 provides the model results by grade cohort and for selected outcome 
measures.® For each grade cohort, results are first presented for the model without controls for 
student, family, school, and classroom characteristics (labeled the “base model”) and then for the 
“fully specified” model which includes all the control variables. For each model, the parameter 
estimates are shown for both the level (i.e., differences in average outcomes between Chapter 1 
participants and nonparticipants) and slope effects (i.e., differences in rates of change over time). 
The cell entries represent the difference between the particular subgroup of students (e g., lst- 
grade cohort with one year of Chapter 1 assistance) and students who did not receive Chapter 1 
assistance in any of the observed years. For example, the first entry for the lst-grade cohort of 
“-34” means that students who received Chapter 1 assistance for one year scored, on average, 34 
scale-score points in Math Applications below the average of those students who never received 
this type of Chapter 1 assistance between the 1st and 3rd grades. 9 Also provided in Exhibit 2.4 
for each cohort is the percent reduction in this “gap” that resulted from adding all the covariates 
to the model (i.e., the percentage difference in level between the base and full models). 

Starting with the base model, Exhibit 2.4 shows that across outcome measures students 
who received any Chapter 1 assistance have lower achievement scores, and lower teacher ratings, 
than students who never received this type of help in school. Moreover, those students who 



8 For brevity, wc have selected a subset of the outcome measures. However, the results were consistent across all of the student 
measures. For complete information see the accompanying Technical Report. 

9 The standard deviations for the CTBS testa range from 40-50 acalc-scorc points. 
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Exhibit 2.4: Summary of Statistical Model Results With Chapter 1 Amount Variables by Grade 
Cohort: Base Model vs. Fully-Specified Model 




Outcome and Years 
of Chapter 1 
Participation 



Base Model 1 



1 st-Gradc Cohort 



Full Model : 



Level 3 Slope 4 Level I Slope 



licahons Scale Score 



3rd-Grade Cohort 



Base Model Full Model 

Percent 

Change T" 

in 

Level Level Slope Level Slope 



Percent 
Change 
in Level 



1 


-34 5 


N.S. 


-29 


N.S. 


-15% 6 


2 


-35 


N.S. 


-28 


N.S. 


-20% 


3 


-36 


N.S. 


-27 


N.S. 


-25% 



-27 


N.S. 


-22 


N.S. i 


-19% 


-45 


2.3 


-37 


N.S. 


-18% 


-47 


N.S. 


-29 


N.S. 


-17% 


-58 


5.0 


-46 


5.1 


-21% 



Reading Vocabulary Scale Score 



1 



2 



3 



4 



Teacher's Math Achievement Ratin 



-41 


N.S. 


-28 


N.S. 


-7% 


-57 


N.S. 


-51 


N.S. 


-11% 


-66 


N.S. 


-62 


N.S. 


-6% 



-20 


N.S. 


-17 


N.S. 


-15% 


-31 


-3.0 


-24 


-4.6 


-23% 


-39 


mm 


-20 


mm 


I 

-23% 


-47 


- 2,1 


-34 


N.S. 


-28% 




1 


-0.24 


-0.10 


-0.19 


-0.08 


-21% 


2 


-0.39 


-0.12 


-0.32 


-0.10 


-18% 


3 


-0.42 


N.S. 


-0.35 


N.S. 


-17% 



-0.34 


-0.05 


-0.27 


N.S. 


-21% 


-0.61 


N.S. 


-0.50 


N.S. 


-18% 


-0.63 


N.S. 


-0.56 


N.S. 


-11% 


-0.65 


N.S. 


-0.56 


N.S. 


-14% 



Teacher's Reading Achievement Ra 



1 


-0.39 


-0.08 


-0.32 


N.S. 


-18% 


2 


-0.60 


-0.11 


-0.50 


-0.06 


-17% 


3 


-0.64 


-0.13 


-0.57 


-0.09 


-11% 




-0.44 


N.S. 


-0.36 


N.S. 


-18% 


-0.71 


N.S. 


-0.54 


N.S. 


-24% 


-0.81 


N.S 


-0.61 


N.S. 


-25% 


-0.88 


N.S. 


-0.66 


N.S. 


-43% 
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r 





1 st- Grade Cohort 


3r< 


1-Grade Cohort 




Outcome and Years 


Base Model 1 


Full Model 2 


Percent 


Base Model 


Full M 


odel 


Percent 
Change 
in Level 


of Chapter 1 
Participation 


Level* 


Slope 4 


Level 


Slope 


Change 

in 

Level 


Level 


Slope 


Level 


Slope 


Reading Teacher's At 


tention Ral 


ting* 








1 


-0.22 


-0.04 


-0.17 


N.S. 


-23% 


-0.22 


N.S. 


-0.18 


-0.007 


-18% 


2 


-0.33 


N.S. 


-0.30 


N.S. 


-9% 


-0.38 


0.04 


-0.29 


0.03 


-24% 


3 


-0.30 


N.S. 


-0.34 


N.S. 


-13% 


-0.36 


N.S. 


-0.30 


N.S. 


-17% 


4 




.. 







— 


-0.26 


N.S. 


-0.29 


N.S. 


+11% 


Math Teacher's Atten 


lion Ratim 


?** 




1 


-0.24 


-0.03 


-0.12 


-0.03 


-50% 


-0.22 


N.S. 


-0.18 


N.S. 


-18% 


2 


-0.40 


N.S. 


-0.19 


N.S. 


-53% 


-0.36 


0.03 


-0.32 


0.009 


-11% 


3 


-0.41 


N.S. 


-0.14 


N.S. 


-66% 


-0.38 


N.S. 


-0.33 


N.S. 


-13% 


4 


- 


- 


- 






-0.37 


N.S. 


-0.22 


N.S. 


- 41% 1 



Data not available 

* This model uses Chapter 1 participation in reading. 

** This model uses Chapter 1 participation in math 

N.S. Not statistically significant at the 0.001 level. 

1 Base model presents multi-level model results without student, family, and school control variables. 

2 Full (or fully-specified) model presents model with student, family, and school control variables. 

3 Level refers to the average outcomes for participants relative to non-participants. 

4 Slope refers to the rate of change in level over time. 

5 Reads, “Students who receive one year of Chapter 1 assistance scored 34 scale score points below non-partiepants on math 
applications (under the base model).” 

6 Reads, “Adding the control variables to the model reduces the difference between participants and non-participants by about 
1 5 percent” 
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received more assistance generally scored below those students who received less assistance (he., 
had fewer years of participation). These results do not necessarily indicate a negative impact of 
the Chapter 1 program. Rather, as previously discussed, they are primarily an indication that the 
right students are being targeted for help. 

It is also the case that, with few exceptions, there were no significant slope effects, and 
where they were found, the slope effects were often negative. 10 This means that either there were 
no differences in growth (i.e., no relative change over time), or where there were differences they 
tended to increase the observed gaps in average performance. What we would have expected to 
see is a positive effect on students’ rate of growth (i.e., the slope effect) indicating that Chapter 1 
assistance was helping close the gaps in student achievement. Again, because of the program's 
inherent selection bias we cannot conclude that Chapter 1 had no effect — we simply do not know 
what would have happened to the participants in the absence of the program, since they are so 
different from nonparticipants and none of those differences have been removed in this first 
model. What we can say, however, is that the help they received was generally insufficient to 
move participating students toward parity with their more advantaged peers (i.e., we found either 
small, insignificant, or negative slope effects). 

What happens when statistical controls 
are added to the model? As also shown in 
Exhibit 2.4, the magnitude of the average (or 
level) differences between participants and 
nonparticipants was reduced with the addition 
of the control variables but did not completely 
disappear. Participants still consistently 
scored below nonparticipants. Furthermore, the differences in achievement among students who 
get more versus less assistance, while generally reduced, persisted in their stratified pattern, i.e., 
students with the most years of help scored at the lowest level. Finally, with the addition of the 



After controlling for student , family . and 
school differences between Chapter l 
participants and nonparticipants , we still find 
that participants score lower than 
nonparticipants and that this gap in 
achievement is not closed over time . 



10 Where numbers are shown in Exhibit 2.4 the differences were statistically significant at the 0.001 level. We have used this more 
stringent criterion due to the large number of statistical tests that were conducted 
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statistical controls, the slopes did not become positive. That is, the negative gap between 
participants and nonparticipants continued over time. 

The statistical models used controlled for a large number of variables and reduced the gap 
between the different groups of students. But, because there was essentially no effect on the rates 
of growth when control variables were added, the trajectories remained parallel over time. In 
other words, students who entered either the 1st or 3rd grade and who were identified to be in 
need of Chapter 1 assistance scored, or were rated, below those who were not targeted for 
assis- ince. Moreover, those students who were targeted for help in multiple years scored below 
those selected for less assistance. Most students, whether they did or did not receive assistance, 
exhibited positive changes over time. However, the rates of growth of Chapter 1 and non- 
Chapter 1 students were similar, indicating that Chapter 1 assistance did not compensate for the 
initial deficiencies of the disadvantaged students. 

Modeling the Pattern of Chapter 1 Participation 

Exhibit 2.5 provides the same type of 
results discussed above but uses the pattern, 
rather than the number of years, of Chapter 1 
services that participants received. To 
minimize the effect of prior participation in 
Chapter 1 we focused this analysis on the 1st 
grade cohort. To help make the results more apparent, Exhibits 2 6 and 2.7 show plots of the 
predicted curves for Reading Vocabulary and Math Concepts and Applications for the fully 
specified” model. (The story to be derived from the graphs for the other outcomes is essentially 
the same.) 

What is so striking about these results is again the “lockstep” relationship among the 
different participation patterns. That is, the initial group differences remained relatively constant 
over time. Even though the addition of statistical controls brought the curves closer together 



Where children start out compared to their 
classmates, largely explains their relative 
academic standing in later grades. Moreover. 
Chapter i assistance did not reduce the initial 
gaps in academic achievement between 
participating and non-participating students 
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Exhibit 2.5: Summary of Statistical Model Results With Chapter 1 Pattern Variables, 1st 
Grade Cohort: Base Model vs. Fully-Specified Model 



Ist-Grade Cohort 



Outcome and Chapter 1 
Participation Patterns 1 



Base Model 


Full Model 


Level 


Slope 


Level 


Slope 




Percent 
Change in 
Level 



100 


-31 


N.S. 


-2° 


N.S, 


-10% 


010 


-31 


N.S. 


-36 


N.S. 


+16% 


001 


-39 


N.S. 


-22 


N.S. 


-44% 


101 


-41 


N.S. 


-26 


N.S. 


-37% 


Oil 


-53 


N.S. 


-34 


N.S. 


-36% 


lie 


-38 


N.S. 


-45 


N.S. 


+ 18% 


111 


-48 


N.S. 


-36 


N.S 


-25% 


Reading Vocabulary Scale Score j| 


100 


-39 


N.S. 


-32 


N.S. 


-17% 


010 


-38 


N.S. 


-46 


N.S. 


+21% 


001 


-50 


N.S. 


-37 


N.S. 


-26% 


101 


-54 


N.S. 


-52 


N.S. 


-4% 


Oil 


-59 


N.S. 


-51 


N.S. 


-14% 


110 


-58 


N.S. 


-50 


N.S. 


-14% 


111 


-66 


N.S. 


-61 


N.S. 


-8% 


Teacher's Math Achievement Rating 



-0.20 


-.14 


-0.22 


N.S. 


+10% 


-0.28 


N.S. 


-0.22 


N.S. 


-21% 


-0.26 


N.S 


-0.13 


-0.13 


-50% 


-0.34 


-.12 


0.33 


N.S. 


-3% 


-0.44 


N.S. 


-0.31 


-3.9 


-30% 


-0.47 


-.16 


-0.35 


N.S 


-26% 






.n-re 




~m% 
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Exhibit 2.5: (continued) 



Outcome and Chapter 1 
Participation Patterns 1 


Ist-Grade Cohort | 


Base Model 


Full Model 


Percent 
Change in 
Level 


Level 


n 


Level 


Slope 


1 Teacher's Reading Achievement Rating 


100 


-0.27 


-0.15 


-0.32 


N.S. 


+19% 


010 


-0.41 


N.S. 


-0.45 


N.S. 


+10% 


001 


-0.52 


N.S. 


-0.24 


-0.12 


-54% 


101 


-0.51 


-0.15 


-0.55 


N.S. 


+8% 


011 


-0.68 


N.S. 


-0.45 


-0.11 


-34% 


110 


-0.66 


-0.14 


-0.54 


N.S. 


-42% 


111 


-0.65 


-0.13 


-0.57 


-0.08 


-12% 


I Reading Teacher's Attention Ratings* 


100 


-0.20 


-0.05 


-0.16 


N.S. 


-20% 


010 


-0.25 


N.S. 


-0.26 


N.S. 


+4% 


001 


-0.21 


N.S. 


-0.12 


-0.07 


-43% 


101 


-0.28 


N.S. 


-0.24 


N.S. 


-14% 


011 


-0.41 


N.S. 


-0.29 


N.S. 


-29% 


110 


0.34 


N.S. 


-0.36 


0.06 


+6% 


111 


-0.31 


N.S. 


-0.34 


N.S. 


+ 10% 


\ Math Teacher's Attention Ratings* 4 I 


100 


-0.16 


-0.08 


-0.11 


N.S. 


-31% 


010 


-0.25 


N.S. 


-0.13 


N.S. 


-48% 


001 


-0.32 


N.S. 


-0.12 


N.S. 


-63% 


101 


-0.34 


N.S. 


N.S. 


N.S. 




011 


-0.48 


+0.05 


-0.18 


N.S. 


-63% 


110 


-0.35 


N.S. 


-0.24 


N.S. 


-31% 


111 






- ni4 


NS 





* This model uses Chapter 1 participation in reading. 

** This model uses Chapter 1 participation in math. 

N.S. = not statistically significant at the 0.001 level. 

1 Patterns are defined by whether a student received Chapter 1 assistance in a particular year. For example, ’000’ = 
never received assistance, “100’ = received assistance in 1st grade, but not in 2nd or 3rd grades. 
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Exhibit 2.6: Estimated Reading Vocabulary Scale Scores, by Grade Level 




Patterns are defined by whether a student received Chapter 1 assistance in a particular year. For example, '000' = never 
received assistance. "100" = received assistance in 1st grade, but not in 2nd or 3rd grades. 




Exhibit 2.7: Estimated Math Concepts and Applications Scale Scores, by Grade Level 
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(i.e., comparing the results for the base and fully-specified models), the lack of an effect on the 
slope of the curves resulted in the persistence of the initial gaps and the stratification by amount of 
assistance over the observed years of schooling. 



What do we conclude from these results? Apparently, exposure to school, and general 
maturation, cause most children to grow over time in their level of achievement. However, where 
they start out has a great deal to do with where they end up relative to other children in their class. 
Further, Chapter 1 assistance does not reduce the initial gaps in achievement between participating and 
non-participating students. 

Analysis Using A Comparison Group 

As discussed above, this stage of our analysis consisted of an attempt to create a group of 
“presumptively eligible” students entering 1 st grade, and to then examine the growth patterns of 
those students in this group who do, and do not, receive Chapter 1 assistance. To reiterate, 
those students deemed to be presumptively eligible were those children who 3) attended schools 
where 50 percent or more of the enrolled students were eligible for free or reduced-price school 
meals, and 2) who scored at or below the 25th percentile on the CTBS standardized achievement 
test at the start of the 1st grade measured in fall 1 991 11 

Three-level HLM models were again estimated for both Reading Vocabulary and Math 
Applications, using (as above) measures of both the amount and pattern of Chapter 1 assistance. 
Because the results for amounts and patterns tell the same story, we have, in the discussion that 
follows, focused only on the pattern variables. Finally, a second modeling approach was attempted to 
extend the comparison group analysis by adding the fall 1991 test scores to the models as an additional 



1 1 We tested the comparability of those presumptively eligible students who did and did not receive Chapter 1 assistance, using 

Ordinary Least Squares (OLS) regression to compare fall 1 991 test scores of the selected 1 st-gradc students who were receiving 
Chapter 1 in spring 1992 to those who were not receiving such assistance. There were no significant differences between the 
two groups for CTBS Reading Vocabulary at that time, and a modest significant difference between the two groups for the 
CTBS Math Concepts and Applications. Thus, this attempt to create comparison groups helped reduce the selection bias for 
Reading Vocabulary, but did not completely eliminate differences for the CTBS math scales. But it is important to remember, 
these are not true randomly selected control groups. They have been created through post-hoc analysis and are, as a 
consequence, still vulnerable to a variety of confounding influences and biases 
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covariate. This was done to further control for initial differences between the Chapter 1 participants 
and nonparticipants. 



Group Matching Alone. Exhibit 2.8 shows the predicted curves resulting from the estimated 
models for Reading Vocabulary. Because of our focus on a selected group of low-achieving and 
economically disadvantaged students, these curves are, on average, lower than those previously 
presented. However, the results are essentially unchanged. That is, all students exhibited growth over 
time, but students who received Chapter 1 assistance scored significantly below those who did not; and 
those who received more help scored lower than those who received less Moreover, because the 
slope effects were again statistically insignificant, we cannot discern any “compensatory” effect over 
time associated with Chapter 1 participation 

Exhibit 2.9 shows similar predicted curves for Math Applications. As with achievement 
scores in reading, all students showed positive growth But. unlike reading, there were no 
significant differences in either average level or slope (growth rate) between participants and 
nonparticipants, and among the different yearly patterns of program participation This model, 
then, reduced selection bias enough so that we no longer find significant negative effects 
However, we still do not discern positive effects from Chapter 1 . 

Initial Test Score as a Covariate. Adding students' initial test scores to the three-level 
model somewhat improved the comparability of the groups of participants and nonparticipants 
However, despite this improvement the results are essentially the same as those discussed above 
(see Exhibits 2.10 and 2.11). The only difference is for Math Applications where we find a small, 
but statistically significant, interaction between grade (i.e.. time) and initial test score at the entry 
to 1st grade In other words, students who scored lower at entry to school, achieve at a slightly 
slower rate than other students. 

The basic conclusion again is that the effect of Chapter 1 participation is independent of 
students' growth rates (i. e. , there are still no positive slope differences between participants and 
nonparticipants). 
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Exhibit 2.8: Estimated Reading Vocabulary Scale Scores, by Grade Level: Comparison Group Analysis 
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Patterns are defined by whether a student received Chapter 1 assistance in a particular year. For example, '000* = never 
received assistance, ’100* = received assistance In 1st grade, but not In 2nd or 3rd grades. 




Exhibit 2.9: Estimated Math Concepts and Applications Scale Scores, by Grade Level: Comparison Group Analysis 
















Exhibit 2.10: Estimated Reading Vocabulary Scale Scores, by Grade Level: Comparison Group Analysis With Fall 1991 Test Score 




Patterns are defined by whether a student received Chapter 1 assistance in a particular year. For example, "000" = never 
received assistance, "100" = received assistance in 1st grade, but not in 2nd or 3rd grades. 




Exhibit 2.1 1 : Estimated Math Concepts and Applications Scale Scores, by Grade Level: Comparison Group Analysis With Fall 1991 Test 
Scores 
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Within Student Analysis 



I A final set o analyses involved estimating simple HLM models (without covariates) to test 

whether scores were higher in years in which students received Chapter 1 assistance compared to 
years in which they did not receive supplemental help. 12 This was done within the following 
subgroups of students: 

■ Ist-grade cohort : a) those who received assistance in one out of three years and, 

b) those who received assistance in two out of three years; and 

* 3rd-grade cohort: a) those who received assistance in one out of four years, b) 

those who received assistance in two out of four years; and, c) those who received 
assistance in three out of four years 

The results did not detect any statistically significant differences from the expected yearly 

growth in test scores between the years in which students did and did not receive assistance. 

These results, then, support what was found in the other analyses, i.e., there was no indication in 

the Prospects data that Chapter 1 participation improved students' educational outcomes. 

I 

I 

i 

l Important Analytical Issues 

i 

i 

I 

I 

| In interpreting these results on the impact of Chapter 1 , two important issues must be kept 

l in mind. First, because Chapter 1 was primarily a funding mechanism, local districts and schools 

i 

| could choose to provide Chapter 1 services to students in a variety of ways. Thus the 

intervention or “treatment” that was the subject of this evaluation did not mean the same thing for 
all participating students, even among students in the same school. For example, some students 
could receive additional instruction in reading, for others the support may have been targeted at 
math skills, and for still others extra help may have been provided in both reading and math. 

These services could also be delivered in a variety of modes and environments: by pulling the 



12 The models were estimated for Reading Vocabulary and Math Concepts and Applications under two different analytical 

scenarios: 1) sampled students were allowed to have their own levels and growth curves (i.c., slopes), and 2) students were fixed 
at avenge slopes but allowed to have varying individual levels. 
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child out of the regular classroom to receive the instruction; by replacing the classroom module 
(e.g., math instruction) with a Chapter 1 module taught in the same time period in a different 
classroom; by a classroom aide or certified teacher helping the child in his own classroom, in a 
before-school, after-school, or summer school program; or in a schoolwide project. In addition, 
Chapter 1 funds could also be used to support preschool, full-day kindergarten, staff 
development, and to purchase computer software and hardware and other educational materials. 
Consequently, Chapter 1 was not a uniform program, but a collection of services and practices 
provided in participating schools using federal resources. 



Further, we cannot assume that funds or services labeled “Chapter 1” represented 
additional resources, or even different services, from those schools or districts would have 
undertaken without Chapter 1 funds. For example, state and local compensatory education funds, 
or general education funds, could have been used to provide the same types of services to non- 
Chapter 1 students as were provided by Chapter 1. Chapter 1 funding was so long established, 
and so endemic to the U.S. public school system, that the allocation of Chapter 1 funds to a 
school district or school may not have necessarily meant that the resources for that district or 
school were increased on net - or even that the funds were used in different ways than they 
would have been used if they were not labeled “Chapter 1 .” 

The second issue of importance is finding an appropriate comparison group from which to 
measure program impacts. The methodological question of paramount importance for this study 
is, “How do we know if Chapter 1 has had a positive impact on students 9 ” The problem of 
attribution of observed outcomes (e.g., student educational progress) to special services (e.g., 
Chapter 1 reading instruction) rather than to other contributing factors (e.g., home environment) 
is at the heart of all research, whether it be in physics, economics, or education policy. Over the 
last 30 years, social scientists have struggled with this problem of determining whether a 
particular program contributed to a desired policy result, i.e. , whether the outcomes observed for 
program participants (e.g., increased reading and math achievement) would have been achieved in 
the absence of the program. This is what is known as the problem of the counterlactual - 
measuring what would have happened absent the program In general, measuring changes for 
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those who received some intervention is relatively easy, but measuring what would have occurred 
had they not received the assistance is not so simple. 



An experimental research approach was not possible for this study. As a consequence, 
Prospects represents a longitudinal study of selected children who did and did not receive 
Chapter 1 assistance, and who were in schools across the full range of disadvantaged 
circumstances . Therefore, it is virtually impossible to establish a true natural comparison group, 
comprising pupils who were exactly like those who received Chapter 1 assistance but who were 
not served by the program. Using primarily the results of standardized achievement tests, districts 
selected the most educationally disadvantaged children for services in those schools that had the 
highest concentrations of poor children. Therefore, naturally occurring comparison groups 
would include either students in the same high-poverty schools who were measurably less 
educationally disadvantaged, or in schools that were not receiving Chapter 1 assistance (at least, 
in the same grade). On average then, a selection of nonparticipants would tend to produce a 
group of students who were marginally less disadvantaged. 

To attempt to deal with this noncomparability, we have used statistical procedures to 
“control for 1 ' measurable differences between participants and nonparticipants. However, 
statistical techniques are only as good as our ability to measure those characteristics that make the 
two groups of students different. And, even with perfect measurement tools, one is never certain 
that all the ways in which students can differ have been accounted for in the analytical model. As 
a consequence, any conclusions about the effect of Chapter 1 that are derived from this study can 
only be considered suggestive. Conclusive attribution is simply not possible within the constraints 
of the study design. 

Summary 



The initial descriptive analysis of the patterns of student achievement for those 
children who did and did not receive Chapter 1 assistance indicated a lack of a “compensatory 
effect. That is, disadvantaged children exhibited growth over time, but they were seemingly 
unable to close the gap in achievement between themselves and their nondisadvantaged peers. 
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The use of sophisticated statistical 
techniques to extend this analysis by 
controlling for measurable differences 
between participants and nonparticipants 
provided a more valid basis from which to 
judge program effects. The key finding from 
these analyses was that controlling for a 
variety of student, family, school, and 
classroom variables did not change the basic 
descriptive conclusions: 



Chapter 1 served those children who were 
clearly in need of supplementary educational 
assistance But the services appear to be 
insufficient to allow them to overcome the 
relatively large differences between them and 
their more advantged classmates. 

This inability to find a compensatory effect of 
Chapter 1 does not mean the program was a 
failure Limitations of this study do not allow us 
to determine whether Chapter 1 students would 
have been academically worse of] without the 
assistance they received. 



Students selected to receive Chapter 1 assistance had test scores, or were rated by 
their teachers, below their nonparticipating classmates; 

Those students who received more Chapter 1 assistance (i.e., more years of 
service) had test scores, or were rated, below those who received less help, and, 

Chapter 1 students did not, over time, close the performance gap between 
themselves and their more advantaged classmates. 



The first implication to be drawn from these results is that Chapter 1 was, on average, 
serving those students who were clearly most in need of supplementary assistance. That is, 
students who received assistance scored below those who did not, and repeated selection for 
compensatory education was associated with greater educational deficits 

Second, Chapter 1 assistance was, on average, insufficient to close the gap, or compensate 
for initial achievement differences between advantaged and disadvantaged students. The observed 
“lockstep pattern” of student growth clearly demonstrated that where students started out, 
relative to their classmates, is where they ended up in later grades 

Finally, our inability to discern a compensatory effect of Chapter 1 is not necessarily an 
indication of program failure. As noted above, the design of the Prospects study does not allow 
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us to observe directly whether Chapter 1 students would have been worse off (i.e., whether the 
gap would have widened over time) in the absence of the services they received. In fact, because 
we might expect the gaps to grow over time, without a special intervention, it may be that 
Chapter 1 is helping participating students but is too weak an intervention to bring them up to par 
with their classmates. 



/56 Prospects: Final Report On Student Outcomes 



Abt Associates Inc. 



CHAPTER THREE 



CHARACTERISTICS OF HIGH-PERFORMING 
HIGH-POVERTY SCHOOLS 



A central concern of the current educational reform movement is the performance of 

students in our nation’s most disadvantaged schools. Students in these high-poverty 
schools generally have the highest rates of dropout from school, and the lowest performance on 
most indicators of school success and involvement. Moreover, there is a growing understanding 
that the nation’s schools cannot be revitalized by attending to only some of our schools and some 
of our children. The most difficult challenge facing educators today is to carry out the mandate 
and promise to educate all children. How do schools respond to this challenge? What practices 
do they use and are they effective for achieving this end ' 1 

The Prospects sample of schools provides an unusually rich source of information to 
describe high-poverty schools generally but also those schools that appear able to produce 
consistently positive outcomes for disadvantaged students. Specifically, this chapter addresses 
three important questions: 

■ How do high-poverty schools differ from low-poverty schools in their 
organizational and instructional structure, resources, and processes 1 

■ How can we define and identify high-performing high-poverty schools 1 

■ What are the characteristics of such high-performing high-poverty schools 1 How 
do high-performing high-poverty schools differ from other high-poverty schools 1 
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A Profile of High-Poverty Schools 



Tables provided in Appendix B compare high- and low-poverty schools in the Prospects 
sample on a range of characteristics. Examining these data provides an interesting portrait of the 
distinctive characteristics of high-poverty schools; 



■ Educational Needs — High-poverty 
schools were more likely to be 
located in urban areas, 
and to enroll a higher percentage of 
minority students and students who 
were severely behind in their reading 
and mathematics performance. 

These schools also had staffing and 
absentee problems that seriously 
curtailed their ability to adequately 
educate students. In particular, high- 
poverty schools had the highest rates 
of teacher and student mobility, and 
the lowest rates of student 
attendance. Consequently, those 
students who needed the most 
opportunity to learn may in fact have received the least amount of actual instruction. 
Finally, although some 90 percent of high-poverty schools received Chapter 1 funds, only 
one-third of them used the schoolwide provision which allows for greater flexibility in 
targeting the use of these additional funds. 

■ School Organization— High-poverty schools were not different from others in terms of the 
number of days in the school year, nor in terms of average enrollment. However, 
reported differences in child/ staff ratios and child/teacher ratios suggest that the additional 
federal resources provided to high-poverty schools were not being used to add more 
teachers, but were instead being used to add aides in the classroom. With respect to 
tracking of students to homogeneous ability classes, there were cohort differences: high- 
poverty schools were more likely to track in grades K-3, about equally likely to track in 
grades 4-6, and less likely to track in grades 7-9 in comparison to low-poverty schools. 

■ School Leadership — Principals of high- and low-poverty schools had comparable years of 
administrative experience, but principals in high-poverty schools had slightly more years of 
education. Decision-making in high-poverty schools was reported to be slightly 



High-poverty schools face greater 
educational challenges than low-poverty' 
schools. These poor schools are generally 
located in urban areas , and enroll higher 
percentages of poor and minority students 
who are severely behind in reading and 
math. These schools are also characterized 
by high rates of teacher and student 
mobility , are more likely to use aides than 
teachers for instruction , have a less 
cooperative school environment \ and have 
access to fewer instructional materials. 
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less collegial, and more centralized. Principals also perceived slightly less support 
from teachers, parents, and the community in high-poverty schools 

■ Parent Expectations and Involvement-Parents of students in high-poverty 
schools had significantly lower expectations for their children’ s educationa 
attainment, and about their children receiving higher-than-average future grades. 
High-poverty schools also had higher levels of parental involvement in home 
educational activities, higher numbers of school contacts, lower levels of 
involvement in school activities, and a lower probability of parents saving money 
for college. Parents of students in high-poverty schools did not have significantly 
different overall opinions of the quality of the school than did parents in low- 
poverty schools. 

■ Classroom Organization and Resources-lhete were no significant differences in 
class size between high- and low-poverty schools, but high-poverty schools were 
more likely to use aides in the classroom. High-poverty schools were also more 
likely to use the in-class model for the delivery of Chapter 1 services, and allocated 
more direct instructional time per week and assigned more homework than other 
schools. Teachers in high-poverty schools were not different in their use of whole 
group instruction. But, teachers in high-poverty schools reported having less 
adequate class materials, and a lower incidence of computer use in their 
classrooms. 

■ Teachers— There were no significant differences in educational background, 
teacher certification, years of experience, or hours of in-service training between 
teachers in high- and low-poverty schools. Teachers in high-poverty schools gave 
less favorable ratings of their school’s climate, and reported lower levels of 
perceived influence over school policy. 

■ Instructional Content/Emphasis — Teachers in high-poverty schools reported more 
emphasis on advanced math skills with no significant differences in the amount of 
emphasis on remedial math skills. For reading, teachers in high-poverty schools 
reported a higher emphasis on remedial and basic skills with no significant 
differences on the emphasis they placed on reading and writing attitudes (the 
development of an appreciation of reading and writing) or the integration of 
reading and writing into other class work. 
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Identifying and Describing High-Performing High-Poverty Schools 



Our effort to identify and define high-performing high-poverty schools involved using the 
HLM models described in Chapter Three to estimate average achievement levels and gains 
(slopes) for all high-poverty schools that also had at least 100 sampled students. From this pool 
of high-poverty schools, “high-performing schools” were then defined as those that had: 1) above 
average student achievement levels (maintained over time) in both math and reading; and/or 2) 
above average levels and growth slopes in either math or reading. Limiting our attention to 
students in the 1st- and 3rd-grade cohorts, we identified three schools in the lst-grade cohort, and 
three schools in the 3rd-grade cohort that met one or both of these criteria. But, because one 
school appeared in both grade cohorts, there were actually a total of five different schools 
identified as high performing. 

Data on the five schools are provided in Appendix C. Although we have already shown 
that, on average, students in high-poverty schools perform poorly on standardized tests relative to 
students in low-poverty schools, the Prospects data also show that there are students in some 
(albeit few) highly disadvantaged schools that perform quite well relative to their more 
advantaged peers— schools we refer to as “high-performing.” However, descriptions of these 
schools must be considered only suggestive and any interpretations should be made with great 
caution. Because of the small sample, we present these data purely as “food for thought” for 
future research on high-poverty schools, and emphasize that no causal attributions can be drawn 
from such descriptions. 

Much like high-poverty schools in general, the identified sample of high-performing high- 
poverty schools served a disadvantaged, diverse, and largely urban population. These schools 
also served a variety of racial and ethnic populations: two of the five high-performing schools in 
both cohorts served a majority Hispanic student body; the remaining three schools served very 
diverse student populations. 
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The high-performing high-poverty schools had generally lower than average student and 
teacher mobility rates Additionally, three of the five high-performing high-poverty schools had 
principals with more years experience than the average high-poverty school. Perhaps more 
significant, all high-performing high-poverty schools had lower than average disciplinary actions 
Consequently, high-performing high-poverty schools appeared to distinguish themselves from 
other high-poverty schools by a more orderly school environment with lower teacher and student 
turnover, and fewer disciplinary problems despite the challenges of educating a disadvantaged 
population. 

All high-performing high-poverty schools received some form of Chapter 1 funding, and 
three of the five schools had schoolwide Chapter 1 programs. In addition, all schools practiced 
tracking of students by ability in the 1st through the 6th grades Schools that elected to provide 
schoolwide services differed in some important regards from other eligible schools. These 
differences are important to keep in mind when considering the role of schoolwide programs in 
creating high-performing schools. First of all, comparing the Prospects schools with schoolwide 
Chapter 1 programs with those that were eligible for this option but did not elect it, shows that 
the schoolwides were, on average, more disadvantaged. In particular, schoolwide programs 
enrolled a higher proportion of poor and minority students, with lower initial test scores, than did 
the high-poverty schools that did not elect to use this program option. 

Moreover, schoolwide status is associated with differences in the operating procedures of 
the schools. The schoolwides appear to have more decision-making centered in the hands of the 
school and the teachers than did other high-poverty schools In addition, there were far fewer 
disciplinary actions per child in the schools that adopted a schoolwide format, and schoolwide 
programs had a significantly lower student/teacher ratio than the high-poverty, non-schoolwide 
programs. 
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High-performing high-poverty schools also distinguished themselves from other high- 
poverty schools with regard to parental support, among other resources. One of the most 
interesting findings is that teachers in the high-performing schools reported a better school 
climate, and better relations with the school administration and the community. Further, 
many of the parents at high-performing high-poverty schools appeared to have higher 
expectations for their child’s future educational attainment. 



Three out of five high-performing 
high-poverty schools also appeared to place 
greater emphasis on instruction in both 
remedial math skills and advanced math skills 
than other high-poverty schools. 

Consequently, these schools appear to have a 
balanced presentation of basic skills and more 
cognitively challenging material. Given the 
ongoing criticism that instruction for 
disadvantaged children is repetitive, 
uninspiring, and limited to basic skills, the 
emphasis on challenging instruction in these 
schools appears to be important. 

The same strategy of challenging 
students seems true in the area of 
reading/language arts. Three out of five of the high-performing high-poverty schools not only 
emphasized remedial skills, but also focused on higher-order comprehension skills. 



There are at least a small number of 
highly disadvantaged schools in which 
students are performing better than in 
other high-poverty schools . These high- 
performing high-poverty schools are 
characterized by: 

■ more experienced principal, 

■ greater use of the schoolwide 
Chapter 1 program , 

■ greater use of tracking by student 
ability , 

■ lower rates of teacher and student 
mobility, 

m a balanced emphasis on remedial 

and higher-order thinking in 
classroom instruction , and 
a higher-levels of community, 

parent, and teacher support for the 
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Summary 



One of the most significant findings from this, albeit limited, analysis is that there are 
indeed schools serving highly disadvantaged children that are consistently performing above 
average on achievement outcomes. On the other hand, there are, in this broad survey, only a 
handful of such schools. Given the small sample size of five high-performing schools found 
here, we reiterate that caution is obviously necessary in any interpretation of the data for only five 
schools. This objection aside, there are some consistent patterns of interest: 



Students of differing races and ethnicities, and living in different regions and types 
of communities, attended high-performing high-poverty schools. Consequently, 
the schools were effective not just because they had particular types of students 
enrolled. 

There were few organizational features of schools that were related to 
effectiveness with the exception of having a schoolwide Chapter 1 program, and 
greater use of student ability tracking. 

There were specific characteristics of principals and staff that were important In 
particular, having a principal with more years of experience and low staff turnover 
were both associated with the identified high-performing schools 

Instructional practices and content emphasis may also distinguish high-performing 
high-poverty schools. Several schools where teachers had adopted a balanced 
view of remedial skills and higher-order thinking had high-performing 
disadvantaged students. Rather than viewing instruction in basic skills as a 
prerequisite for higher-order and more challenging materials, teachers in these 
schools appeared to generally challenge their students with cognitively demanding 
material. 

Finally, community, parent, and teacher support for the mission of the school 
characterized some high-performing schools. It is not clear from our data alone 
whether support causes success, or is the consequence of it. 
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CHAPTER FOUR 



EXPLORATORY ANALYSIS OF FACTORS 
ASSOCIATED WITH STUDENT EDUCATIONAL 
OUTCOMES 

S ince the appearance of the well-known Coleman report in 1966 (Coleman, et 
al., 1966), the effect of schools on student outcomes has remained a 
controversial topic. The most widely debated finding of the Coleman report was that there were 
few differences between schools in the resources and facilities available to students, and that these 
aggregate school characteristics had little to do with student performance once family background 
and student body composition were considered. The report also found that there were 
appreciable differences in test performance as students entered school, and that these differences 
were maintained throughout the years of school. These findings were taken to mean that because 
school-to-school variation in inputs were small, schools could contribute little to closing the initial 
achievement gap. However, as subsequent research has suggested, differences among schools are 
related to practices within schools that do matter for student growth. New literature on effective 
schools has identified processes within some schools that contribute to success with 
disadvantaged students One of the characteristics found to be important is a school’s culture that 
supports the belief that all students can achieve at a high level and that academic achievement is 
the highest priority of the school. 

Prospects contributes to this ongoing debate by providing a rich source of data on a large 
longitudinal sample of students across a wide grade span. This chapter presents an exploratory' 
analysis of the relationships between student, family, and school characteristics and the same 
student educational outcomes examined in Chapter Three. It is important to point out, however, 
that this analyses is based on the same HLM models that were discussed in that earlier chapter. 
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As a consequence the results described below need to be interpreted with some caution We 
estimated statistical models with a large number of independent variables because we were 
primarily trying to control for measurable differences between Chapter 1 participants and 
nonparticipants. As a consequence, any interpretation of a relationship between a single variable 
and a particular student outcome is contingent upon all the other variables that were included in 
the model (the estimated effects are referred to as “marginal” associations). Because our primary 
purpose was to estimate the effect of Chapter 1 on student outcomes, our statistical models are 
not necessarily optimal for exploring a specific relationship between some construct of interest 
(e.g., parental involvement in children’s education) and student educational outcomes. 

Student vs. School Differences 



One of the more striking results of the 

multivariate analysis of the Prospects data is that 

variation in student outcomes is much la ?,er within 

schools than between schools. For example, for 

standardized tests administered to 1st grade 

students in reading, 59 percent of the observed 

variation is attributable to within school 
* 

differences among students, and 17 percent is 
associated with differences between schools (the 
remaining variation is associated with individual 
student variation over time). These results are 
essentially the same as those reported earlier by 
Coleman, et. aL, (1966). 

As a consequence, these data clearly show 
that variation in student characteristics, including 



Differences in academic achievement 
within schools is far greater than 
average differences between schools . 
Consequently, variation in student 
characteristics accounts for most of the 
observed differences in student 
achievement and classroom 
performance. 

But, these data also show that all schools 
have students who exhibit a range of 
academic achievement levels; poor 
schools are not filled solely with low- 
achieving students , nor are low-poverty 
schools filled solely with high-achieving 
students. There are good students in all 
schools, even in the most troubled 
places, and even the best schools have 
students for whom academic work is a 
constant challenge. 
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family background, account for most of the observed differences in student outcomes. Even so, 
differences in schools clearly do have an important effect on student achievement and classroom 
performance. Furthermore, almost all schools have students who exhibit a range of academic 
achievement levels; poor schools are not filled solely with low-achieving students, nor are low- 
poverty schools filled solely with high-achieving students. There are good students in all schools, 
even in the most troubled places, and even the best schools have students for whom academic 
work is a constant challenge, This finding counters the conventional wisdom and highlights the 
importance of teaching to the needs of individual students wherever we as educators encounter 
them. 

Correlates of Educational Growth 

The next obvious question to ask is, “What factors are associated with higher or lower 
levels of student achievement and rates of growth over time?” To address this question we 
| examined the estimated multivariate models to determine which variables were found to have a 

I statistically significant relationship with student outcomes (as measured by CTBS test scores as 

f well as teacher ratings of student achievement and classroom behavior). Looking across the three 

grade cohorts, and across the different outcome measures, we then identified consistent and 
robust associations that are described below. 

Student and Family Demographics 

Students come to school with a variety of strengths and weaknesses, and family and 
contextual characteristics, that are highly correlated with their success in school. In fact, as noted 
above, differences in student characteristics explain about 59 percent of the overall variation in 
student achievement test scores. As important as these factors are, however, they can be viewed 
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as essentially fixed variables that are beyond 
the influence of schools and teachers. Parents 
send the best children they have to school, and 
it is the responsibility of the educational 
system to help them learn. 

Consistent relationships of student and 
family characteristics on student achievement include the following: 



Student and family characteristics found 
to be highly correlated with academic 
achievement include : age relative to 
classmates (in the early primary grades), 
race and ethnicity , gender, English 
language proficiency, student mobility , 
family income , and parental education. 



■ A student’s age relative to his/her classmates is associated with test outcomes 1 In 
the 1 st cohort, children who were younger than their classmates performed less 
well on standardized achievement tests. However, by 3rd grade the observed age 
differences disappear, i.e., there are no differences between younger and older 
students. 

■ In all three cohorts, student race and ethnicity are strongly related to all measures 
of student achievement, teacher ratings of academic competence, and behavioral 
outcomes. 

■ Gender is a significant factor in determining achievement and teacher rating 
outcomes in all three cohorts. Early in their school career, girls have an advantage 
in vocabulary development, but this advantage is not maintained over time. Across 
all cohorts, girls perform less well on the standardized math test and on the teacher 
ratings of math ability. Teachers, however, rate girls as being more attentive in 
math as well as in reading. 

■ Students with visual, hearing, speech, orthopedic or other physical disabilities 
generally have lower teacher ratings of their academic and behavioral competence, 
and lower standardized test scores, than do children without such disabilities. 

These findings are consistent across all grade levels. 

■ Students with limited-English proficiency (LEP) consistently have lower average 
scores on the standardized tests in reading and math across grade levels than do 
students who are not limited in their English language ability. Teachers also tend 



1 There ire many reasons for variations in the age of children in the same grade, including district policies concerning required ages for school 

entry, parental decisions to enroll or delay school entry for a child, and school practices related to retaining and accelerating (i.e., grade skipping) 
students. 



q ~ 
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to rate these students lower in their reading competence; and teacher ratings are 
also lower for math competence but only for Ist-grade cohort students. 

■ Student mobility (as measured by the number of household moves during our 
survey period) generally has a negative relationship with standardized achievement 
test, and teacher achievement and behavioral ratings in the lst-grade cohort 
Mobility was not related to vocabulary development, however. In the 3rd-grade 
cohort, there are fewer significant relationships between mobility and student 
outcomes (and these relate only to vocabulary test scores and teacher ratings of 
reading ability). 

■ Students with health problems (as indicated by their teacher) were rated lower on 
their academic abilities and their behavioral outcomes than students with no health 
problems. 

■ Total family income and SES (a composite measure of parental education and 
occupation) are consistently, and positively, related to student achievement test 
scores and teacher ratings, i.e., higher income and SES are associated with higher 
levels of performance. 

■ Preschool attendance had a positive effect on teacher ratings of academic 
performance and behavior, as well as on standardized achievement test results for 
students in the 1 st cohort. (Reliable data were unavailable for the 3rd-and 7th- 
grade cohorts.) This is an important finding that supports the recent increased 
focus on early childhood education. 

■ Having a single parent (in most cases having only one adult in the household) was 
generally negatively related to teacher ratings and standardized test scores. 

Children from larger families also performed at a lower level. Both of these 
relationships were, however, inconsistent across the grade cohorts. 



Parent Involvement 

A second category of characteristics is related to parents’ involvement in the education of 
their children. To begin with, parental expectations for their children’s educational attainment 
are consistently related to school achievement. Parents’ assessment of whether their children will 
attend college or not, and whether they will receive better than average school grades, are both 
positively and significantly related to scores on standardized achievement tests, and teacher 
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achievement and behavioral ratings. Parents’ 
expectations for whether their child will 
complete high school are also positively 
related to student outcomes. 

Various scales were also developed 
in conjunction with the National Goals Panel. 

The first scale focused on home activities 
and monitoring. Parents indicated the 
frequency with which they were involve: ; 
such activities as reading to the child, helping 
their child with homework, talking with the 
child about his /her problems or worries, or 
checking to see if homework was completed. A total of twelve such items were used to construct 
this composite measure. Students at all grade levels, and for all outcomes, who had parents that 
were more actively involved in these sorts of home activities did less well than students whose 
parents were not so involved with their children. At first glance, these results seem puzzling and 
contradictory: how can more involvement of the parent be related to lower performance and less 
favorable teacher ratings? It is impoitant to note, however, that many of the items in this 
composite ask about activities that comprise monitoring and direct supervision from the parent 
One interpretation of these results, then, is that parental involvement as measured by this 
composite is a response to students who require greater attention and supervision because they 
have lower achievement scores. 



Parents ’ expectations for their children *s 
success in school are positively related to 
school achievement. 

Parents who were more involved with their 
children 's school work were associated with 
lower levels of student academic 
achievement . This reflects the greater need 
for attention and supervision for low- 
performing students. 

Parents who were more actively engaged in 
their children 's school had children who 
consistently exhibited better academic and 
behavioral outcomes. 



Promoting parent involvement with school is a prominent goal in current systemic school 
reform efforts, particularly in the new Title I changes. Three aspects of this objective are 
captured by the Prospects data: 
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School Contacts With Parents. The first measure is the extent to which parents 
are contacted by the school for a variety of reasons, such as the child’s attendance, 
health issues, and disciplinary matters Both negative and positive reasons for 
contact by the school were included in this scale Not surprisingly, the frequency 
of contacts by the school is negatively related to all student outcomes in the 
different grade cohorts (i.e., more contact is generally associated with lower 
performance). 

Parent Participation in School This second scale captures the frequency with 
which parents participated in such school activities as parent -teacher association 
meetings, a school event in which the child was involved, and as a volunteer for 
school projects. Parents who participated more often in the life of the school had 
children who consistently performed better on the standardized test, and were 
rated more academically able and attentive by the teachers. 

Parent Perception of School Quality. Parents assessed their child’s school in 
terms of the extent to which they thought the school placed a high priority on 
learning, whether they thought their child was challenged at school, and whether 
they believed they had a say in the setting of school policy. Positive parental 
ratings of school quality were consistently related to positive school outcomes for 
children across all outcomes and cohorts. 



School and Teacher Factors 



School and Teacher Factors comprise the final area of interest that is associated with 
student outcomes These variables are clearly the most important as they represent characteristics 
that can be directly manipulated through school policy and program change. Key findings include 
the following: 



* Repeating (or skipping) a grade has been 
frequently, and often heatedly, debated. 
When retained students are compared to 
nonretained students at the same age (i.e., 
retained students’ 1 st-grade scores are 
compared to nonretained students’ 2nd- 
grade scores), retained students are found 
to have lower test scores than their age 
mates 
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■ Class size shows an inconsistent and unexpected relationship with student 
outcomes. For example, in the lst-grade cohort, larger class size is positively 
related to reading outcomes. It is clearly counterintuitive to find that students in 
larger classes perform better than students in smaller classes, but this result may be 
appropriate. High-poverty schools have frequently reduced class sizes, especially 
for reading instruction in the early grades, in an effort to target their resources at 
an important educational stage. One interpretation then is that high-poverty 
schools have reduced class size, but achievement in these schools is lower in 
comparison to more advantaged schools. In other words, even with reduced class 
size, performance continuer to lag at high-poverty schools 

■ The presence of classroom aides during math instruction is positively linked to 
math achievement on the standardized test, but not to the other student outcomes. 

■ The availability of computers in reading instruction appears to have positive 
effects on standardized achievement outcomes, ratings of competence and 
attention, but only for the students in the lst-grade cohort Interestingly, similar 
relationships were not found for math instruction 

■ Instructional time was not consistently related to test scores, nor to the teacher 
ratings of student behavior. 

■ Instructional emphasis 1 in reading/language arts and English was measured by 
creating six composite factor scales: emphasis on remedial instruction; emphasis on 
comprehension; emphasis on writing attitudes and skills; emphasis on reading 
attitudes; emphasis on the application of skills in reading; and emphasis on the 
integration of reading and writing. 

An emphasis on remedial topics (e.g., reading readiness skills, following directions) 
wa negatively related to test scores and teacher ratings Conversely, an emphasis 
on comprehension, the development of writing skills, and on an appreciation for 
the importance of writing was positively related to student achievement and 
teacher’s rating of attention and reading competence. 

These results suggest that students in classrooms that have a greater emphasis on 
comprehension and writing, and less emphasis on remedial instruction, perform 
better on the standardized vocabulary test, and on the teacher performance and 
behavioral ratings. Of course, despite our best efforts to control for selection 
factors, in studies of a correlational nature selection bias influences results to an 
unknown degree. Teachers who have students that are more able may also be 



2 Prospects provides only very limited information on classroom instruction depending primarily on teacher reports of classroom practices 
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more likely to focus on comprehension, that is higher ability levels in the classroom 
may facilitate the presentation of higher-order thinking skills. 

Instructional emphasis in math In the lst-grade cohort, an emphasis on 
remedial instruction is positively related to math test scores and to teacher ratings 
of attentiveness. For the 3rd grade, an emphasis on remedial skills is somewhat 
negatively related to standardized achievement scores. An emphasis on advanced 
math is positively related to student test scores in the 3rd-grade cohort, and to 
teacher ratings in the 1 st and 3rd-grade cohorts (i.e., teachers who rate their 
students as highly competent in math are more likely to provide instruction in 
advanced math and to rate their students as paying more attention in math). 

A variety of other classroom measures were also examined including: student- 
teacher ratio; measures of the relationship between the teacher and the school 
administration, community, and staff; extent of perceived support and 
influence; and staff development time Although these factors are important, 
especially in current discussions of school reform, the overall pattern of results 
show inconsistent associations with student outcomes. Clearly, a more detailed 
analysis of the importance of these factors needs to be carried out in the future. 

The poverty level of the school (ove r and above the economic status of an 
individual student) is negatively related to standardized achievement scores. In 
contrast, the proportion of black and Hispanic students has inconsistent 
associations with student outcomes. 

School size, the number of days in the school term, and the practice of tracking 
by assigning students to classrooms on the basis of similar ability are considered to 
be important dimensions of school administration and organization. However, 
these factors were found to have limited and inconsistent connections with student 
outcomes. 

The number of disciplinary actions reported in the school exhibited a relatively 
consistent relationship to achievement and behavioral outcomes for 3rd grade 
students. Students in schools that had higher numbers of disciplinary actions had 
lower scores on the standardized tests, and had teachers who were less likely to 
favorably rate their academic performance (there were no differences for the 
attention ratings) 

Teachers’ perceptions of community and school support showed inconsistent 
results, being related to teacher ratings for the 1st grade only. 

The decision making practices in the school, which are of some interest to 
current school reform efforts, were found to be related to the achievement 
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outcomes, but in an inconsistent fashion. Teacher ratings of attention in reading 
for the 3rd-grade was positively linked to this factor; for the 1 st-grade cohort it 
had a positive effect for teacher ratings of reading only 



Summary 

The disparate findings discussed thus far tell an important story To begin with, student 
differences account for most of the variation in student achievement outcomes. The most 
consistent predictors of achievement are of the following three types: 1) those that are fixed (e.g., 
gender, age relative to classmates), 2) those that are associated with poverty (e.g., race/ethnicity, 
SES), and 3) parental expectations and involvement in school. Of these, only the extent to which 
parents are actively engaged in their child's school, and their expectations for his/her future 
success in school, are amenable to change through educational reform efforts. The dominant 
importance of child and family factors in predicting student achievement outcomes does not mean 
that school factors, beyond the noted opportunity to target parents, are inconsequential. A variety 
of other factors are also consistently related to student educational growth, including student 
retention, the experience and skills of instructional staff, and the emphasis that teachers place on 
comprehension and high-order thinking skills. 
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Exhibit B.l: Means of School-Level Variables in High-Poverty Schools Compared to Low- 
. Poverty Schools (lst-and 3rd-grade Cohorts) 




Percent Poverty } t-Test of 



Variable 



/ItJttcatfoaal Load 



Percent Poverty 
J>75% 
N=446 

Mean (Standard 
Deviation) 



<L25% 

N=400 

Mean 

(Standard 

Deviation) 



Difference 
High vs. Low 
*=P<.05 

**~p<,01 
***ssp< .001 



Percent School Poverty 


88.7 (8.2) 


12.7 (7.2) 


Race: % Black 


39.6 (35.7) 


5.1 (8.1) 


% Hispanic 


35.3 (35.1) 


6.6(12.4) 


% White 


20.6(24.1) 


82.6(18.5) 


% Other 


5.4 (9.1) 


6.7 (10.8) 


Urbanicity: % Urban 


69.0 (46.0) 


35.0 (48.0) 


: Suburban 


18.0(38.0) 


44.0 (50.0) 


: Rural 


14.0(34.0) 


22.0 (41.0) 


Region: % Midwest 


6.0 (23.0) 


21.0 (40.0) 


% Northeast 


22.0(42 0) 


26.0 (44.0) 


% South 


49.0(50.0) 


26.0 (44.0) 


% West 


23.0(42.0) 


27.0 (45.0) 


Percent below 25th percentile m 
Reading 


30.2(16.3) 


10.6 (8 4) 


% LEP Students 


18.6(23.4) 


2.0(5 4) 


Percent Student Mobility 


19.0(12.3) 


7 7 (6 4) 


Percent Teacher Mobility 


8.0 (27.0) 


2.0(12.0) 


Percent R.eceiving Chapter 1 Funding 


90.0 (30.0) 


29.0 (45.0) 


Percent Schoolwide Chapter 1 


33.0 (47 0) 


0.00 (0.05) 


Average Number of Disciplinary 
Actions/Student 


0.13 (.22) 


0.09 (.14) 








50.1 (28.8) 



20.6 (27.0) 



19.2 (25.9) 



54.7 (33.0) 



6.2(10.4) 



49.0 (50.0) 



28.0 (45.0) 



23.0 (42.0) 



12.0 (33 0) 



20.0 (40.0) 



42.0 (49.0) 



26.0 (44 0) 



19.9 (13.5) 



8.6(15.8) 



14 0(10 8) 



4.0 (20) 



66.0 (47.0) 



8.0(28.0) 



0.12 (.18) 
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Exhibit B.l: Continued 




Vcrctmt Poverty 
N «446 

Mean {Standard 
Beviatio n) 



Percent Poverty < 1 t-Teit of Biffermce 



25% 

N«400 

Mean {Standard 
Deviation) 



High v** Low 



Attendance Rate 



SriiotdOrg&nfcatkm 



Number School Days 



School Size 1 



Child/Staff Ratio 



Child/Teacher Ratio 



Use of Tracking 3 : K-3 



Use of Tracking: 4-6 



Use of Tracking: 7-9 






Decentralized Decision Making 3 



Lack of Support 4 



Coordinate Chapter 1 w / Regular 
Instruction 5 



Principal’s Education 6 



Principal’s Experience: Number of 
Years at School 



Principal’s Experience. Total Years 
NS“ Nc4 significant 



92.0 (6.4) 



1.6 (.27) 



2.5 (.68) 



4.8 (.81) 



5.9 (5.1) 



11.3(6.9) 



95.2(3.6) 




6.5 (5.4) 



12.1 (7.3) 



Baaed on categories of total enrollment - higher numbers indicate larger enrollment. 

The proportion of schools that use tracking in this grade span 
A multi-item scale; higher numbers indicate more decentralized decision-making 
A multi -item scale; higher numbers indicate a less supportive environment. 

Based on categories of frequency of staff coordiruiion, e.g., 0 1 = Daily. 02 = Weekly, 03 = Monthly. 
Based on alegories of educitional UUuunent, e .g„ 01= less than B A; 04 = Mastets; 06 - Doctorate. 
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Exhibit B.2: Means of Student and Instructional Variable# in High-Povertv Schools Compared to Low-Poverty Schools 

flat-andS 3rd Grade Cohorts) 



Variable 


Perceni Poverty 
V7S*/a 
N=403 

Mean \ Standard 
Devotion) 


Percent Poverty 
12$% 
N= 5 I032 

Mean ^Standard 
Deviation \ 


of Difference 
v*. Low 

**“p<*0i 


AH Sample 

tmr- 

K^im. 

, 

{FMmiurd 

Dtvbrtfon) 


“amily/ChBd- 

\tmih7Schoal 


i 








Percent of Parents Who Saved Money 
for College 


38.0(41.) 


57.0 0(45.0) 


*** 


45.0 (43.0) 




Percent of Parents Wo Expect Child 
to Graduate College 


52.0(42.0) 


74.0 (40.0) 


*** 


59 0 (43.0) 




Percent of Parents Who Expect Future 
Grades to be Above Average 


34 0 (40.0) 


53.0(44.0) 


* ♦ * 


41.0 (42.0) 




Percent of Parents Who Expect Child 
to Graduate High School 


82.0 (33.0) 


96.0(18.0) 


*** 


88.0 (28.0) 




Parent Involvement: Frequency of 
Home Activities 1 


1.67 (.27) 


1.78 (.38) 


+** 






Average Number of School Contacts 
With Parent 


1.38 (.17) 


1.33 (.21) 


*** 


1.36 (.18) 




Parents’ Opinions of School 2 


1.81 (.30) 


1.79 (.29) 


NS 


1.81 (.29) 




Parent Involvement: Frequency of 
School Activities 3 


1 53 (.24) 


1.63 (.29) 


*** 


1.57 (.27 ) 





NS= Not Significant 



BEST COPY AMILABU 



A mufti-item scale of different home educational activities; lower numbers indicate greater frequency ofparenl-chiid mler action. 
A multi-item scale: lower numbers indicate a more favorable opinion of the school. 

A multi-item scale of different types of school activities; higher numbers indicate more frequent partiepation 
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Exhibit B.2: Continued. 





, :1.1am room Or^anization/Re-soum** 



Class Size 



Percent Chapter 1 Students /Class 



Percent Chapter 1 Students Served In 
Class 



Percent Chapter 1 Students Served 
Out of Class 



Aide in Class 



Percent Whole Group Instruction 



Average Hours of Assigned 
Homework Per Week 



Average Instruction Time 
fMinutes/Week') 



Percent of Teachers Who Had 
Adequate Class Materials 



Percent of Teachers Who Use 
Computers in Class 



Tocher 



Years of Teaching Expenenee 



Teacher Education 4 



Hours of In-Service Irauung/Y ear 



Teacher Certified (1 - Yes) 



Percent Poverty t-Test of 



Percent Poverty 
>75% 
N=4Q3 
Mean 
(Standard 
Deviation) 



3.25 (.66) 



: 82 (01) 



0.92 (.21 1 



<25% 

N-1032 

Mean 

(Standard 

Deviation) 



Difference 
High vs. Low 
*— p<.05 

****p<,001 



All Sample 
Schools 
(Cohort 
1&3) 
N=I43S 
Mean 
(Standard 
Deviation) 



23.7 (5.17) 



20.9 (19.6 1 



13.2 (19.2) 



9.75(11 71 




0 95 (.17 i 



2.86 (. 66 ) 



0.94 (.17) 



BEST CQFV avail-' 



Based on categories of educational attainment, c.g., 01 = less than a BA; 04 ~ Masters; 06 - Doctorate. 
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Exhibit B.2: Continu'd 



Variable 





|>ehool Climate: Relation? 
\wi Administration 1 



Community Climate 2 



Teacher Influence 3 



'Opportunity to Utmw; 

\ EmphsuJs* 



Remedial Skills 



I Higher-Order Thinking 



Advanced Math 



Math Attitudes 



I Opportunity to Leam; Reading 
i Emphasis ^ 



Remedial Skills 



Comprehension 



Application 



Reading Attitudes 



Writing Attitudes 



Reading- Writing Integration 



Percent Poverty 
>75% 
N=403 

Mean (Standard 
Devisrtion) 


gggsg-***" — n 

Percent Poverty 
<25% 
N-1032 
Mean 
(Standard 
Deviation) 


4.49 (.81) 


4.74 { 84) 


4.45 (.66) 


4 78 (.67) 


3.75 (.89) 


4.15 (.86) 


I;.;. | 


2.54 (.29) 


2,52 (.24) 


2.50 (.27) 


2.48 (.25) 


1.19 (.37) 


1.11 (.36) 


2.46 (.25) 


2.48 (.22) 


'liipnRa pMapwi 

mm : rnmmmm 


2.36 (.27) 


2.21 (.29) 


2.58 (.26) 


2.50 (.29) 


1 .64 (.44) 


1.52 (42) 


2.86 (.21) 


2.83 (.29) 


2.21 (.46) 


2 10 (.54) 


2.21 (.42) 


2.23 (.43) 



Difference 
High vs. 



All Saittpk 
Schools 
{Cohort 
l&S) 
H-I43S 
Mean 
(S&iarfard 
Deviation 



4.59(82) 



4.59 (69) 



3.95 (.89) 




2.85 (.23) 



2.15 (.52) 



2.21 (44) 



Multi-item scale; higher numbers indicate better relationships 



Multi-item scale; higher numbers indicate better cooperation. 



Multi-item scale; higher numbers indicate greater influence over school policy 
Four multi-item scales; higher numbers indicate greater instructional emphasis. 
Six multi-item scales; higher numbers indicate greater instructional emphasis. 
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Appendix C 

Characteristics of High-Performing 
High-Poverty Schools 
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Exhibit Cls Comparison of Schooi-Leve) Characteristics, High-Performing High-Poverty Schools and AU High- 

Poverty School * 1 






Variable 



DCCATIONAlvLOAD 



Percent School Poverty 



Race: 



Cohort i 



A 



B 



C 



% Black 



% Hispanic 



% White 



% Other 



Urbanicity: 2 % Urban 



% Suburban 



% Rural 



Region:' 



% Midwest 



% Northeast 



% South 



% West 



Percent Below 25th Percentile: 
Reading 



Fercent LEP Students 



Percent Student Mobility 



Percent Teacher Mobility 



89.7 



0.00 



98.3 



1.8 



0.8 



19.7 



39.0 



7.2 



0.0 



76.0 



28.8 



25.6 



40. 



5.6 



18.0 



3.9 



0.0 



86.3 



1.3 



97.0 



1.3 



0.5 



30.0 



25.0 



8.4 



0.0 



Cohort 3 



D 



85.0 



15.0 



28.0 



E 



F 



76.0 



28.8 



25.6 



46.0 



11.0 



20.0 



4.0 



5.3 



0.0 



40.1 



5.6 



18.0 



3.9 



0.0 



79.0 



43.3 



20.5 



14.8 



11.4 



0.00 



23.0 



7.0 



20.7 



0.0 



AB High- 
Poverty 
Schools 
N *446 
Mean 
{Standard 
Deviation) 



88.7 (8.2) 



39.6 (35.7) 



35.3 (35.1) 



20.6 (24.1) 



5.4 (9.1) 



.69 (.46) 



18 (.38) 



.14 (.34) 



.06 (.23) 



.22 (.42) 



.49 (.50) 



.23 (.42) 



30.2(16.3) 



18,6(23.4) 



19.0(12.3) 



.08 (.27) 



1 Schools A and B/E (this is the school that appearod in both student cohorts) had above average achievement levels maintained over time in 
both math and reading. We reftr tottoe as “high-level schools.*’ On the other hand. School C in the 1st- grade cohort had higher than average 
gains (or ilope) and level in math, and higher than average level in reading. We refer to this site as a “high-tlopc a;wl high-level school ” 
Similarly, School D in the 3rd-grade cohort is a high-level school,” and School F is a high-slpoe and high-level school.” 






2 1 - Yes; 0 « No. 
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Exhibit C.l: Continued 




Cohort 3 



Schoolwide Chapter 1(1= Yes) 



Average Number of Disciplinary 
Actions/Student 



Number School Days 



Attendance Rate 



SCHOOL ORGANIZATION 



School Size 



Child/Staff Ratio 



Child/Teacher Ratio 



Use of Tracking: K-3 (1 = Yes) 



Use of Tracking: 4-6(1 = Yes) 



Use of Tracking: 7-9 (1 = Yes) 



[GOALS. POLICIES, 
.‘LEADERSHIP 



Decentralized Decision Making 



Lack of Support 



Coordinate Chapter 1 w/ Regular 
Instruction 



Principal *s Education 



Principal’s Experience: Years at 
School 



Principal’s Experience: Total Years 




All High*- 

Poverty 

Schools 

Me»» 

(Standard 



8.7 11.7 12.8 
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F a m 0 y/ Chi 1 d- Fara ih/Schoo) 



Percent of Parents Who Saved Money 
for College 



Percent of Parents Who Expect Child 
to Graduate College 



Percent of Parents Who Expect Future 
Grades to Be Above Average 



Percent of Parents Who Expect Child 
to Graduate High School 



Parent Involvement: Frequency of 
Home Activities 



32.0 46.0 26.0 44.0 t 46.0 44.0 

69.0 44.0 64.0 50.0 44.0 54.0 

59.0 34.0 32.0 34.0 34.0 17.0 

89.0 86.0 73.0 83.0 86.0 81.0 

1.65 1.78 1.58 1.93 1.78 1.88 



Average Number of School Contacts 1,34 1.26 1-37 1.31 1.26 1.18 

1.72 1.91 1.55 1.80 1.91 1.81 



Parents’ Opinions of School 



Parent Involvement: Frequency of 



1.66 1.46 1.43 B 1.51 1.46 1.30 



38.0 (41.0) 



52.0 (42.0) 



34.0 (40.0) 



82.0(33.0) 



1.67 (.27) 



1.38 (.17) 



1.81 (.30) 



1.53 (.24) 



} 

Orgaaijafian/RtitmittK 






my 




■Si 




— 1 


Class Size 


20.8 


22.6 


22.1 


17.5 


22.6 


25.7 


33.9(5.1) 


Percent Chapter 1 Students /Class 


19.8 


15.1 


61.2 


19.0 


15.1 


15.9 


33.9(22.6) 


Percent Chapter 1 Students Served In 
Class 


18.0 


16.2 


51.5 


11.6 


16.2 


0.00 


23.9(24.4) 


Percent Chapter 1 Students Served 
Out of Class 


2.45 


0.25 


6.04 


11.1 


0.25 


10.4 


14.2(15.0) 



•-dat* mining 
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Exhibit C.2: Continued 




School Climate: Relations 
w/Administration 

Community Climate 

Teacher Influence 



4.61 4.97 4.52 5.01 4.97 

4.52 4.93 4.44 4.97 4.93 

4.00 3.12 2.73 3.99 3.12 



4.69 (.69) 



4 49 ( 81) 
4.45 (.66) 
3.75 (.89) 
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Exhibit C.2: Continued 



Cotort i 



Cohort 3 





Remedial Skills 



Higher-Order Thinking 



Advanced Math 



Oportunitj to Learn: Hiding 
Emphasis 



Remedial Skills 



Comprehension 



Application 



Reading Attitudes 



Writing Attitudes 



Reading- Writing Integration 




Poverty 

> 75 % 

N-403 

Mean 

{Standard 

Deviation) 



2b0 


2.78 


2.55 


2.60 


2.40 


2.54 (.29) 


2.50 


2.61 


2.52 


2.50 


2.10 


2.50 (.27) 


1.44 


1.42 


1.49 


1.44 


0.98 


1.19 (.37) 


2.69 


2.61 


2.43 


2.69 


2.03 


2.46 (.25) 




2.36 (.27) 



2.58 (.26) 



1 64 (.44) 



2.86 (. 21 ) 



2.21 (.46) 



2.21 (42) 
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